2024-09-13T11:25:57.559 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T11:25:57.576 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T11:25:57.656 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428 branch: wip-aclamk-cbt-combined description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_5925} email: akupczyk@redhat.com first_in_suite: false job_id: '7903428' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2: false bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 5394 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 subset: 582/12160 suite: rados suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi005.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDISKxw6WkgLhz0RS4rY2ZEAPyeTs79c0ri10Vt78mSl3dy7EfYqxAu7uwp4LJpcpDKTdIQnFgFAlmrY/7LR1Jc= smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIIbiBSVAehJs0yx6kwab3IYTXeLbYBX0bMHVgRYoyNJbLdpH8MO1b+709X5587ggt4/22BDGRPvKfVW1TaMP9Q= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 thrash_many: true - exec: client.0: - ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_09:37:56 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T11:25:57.656 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T11:25:57.657 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T11:25:57.657 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T11:25:57.659 INFO:teuthology.task.internal:Checking packages... 2024-09-13T11:25:57.681 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T11:25:57.682 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T11:25:57.682 INFO:teuthology.packaging:ref: None 2024-09-13T11:25:57.682 INFO:teuthology.packaging:tag: None 2024-09-13T11:25:57.682 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T11:25:57.682 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:25:57.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T11:25:57.824 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:25:57.825 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T11:25:57.833 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T11:25:57.833 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T11:25:57.869 INFO:teuthology.task.internal:Saving configuration 2024-09-13T11:25:57.882 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T11:25:57.890 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T11:25:57.909 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:19:21.072869', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEsKFp52ilDaWJykxZmEBjFftTbJFXQ+tqL4l+gBYQmh'} 2024-09-13T11:25:57.937 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:19:21.070221', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB8HhDIp/Yv6Fra/PJyNmhsVwnp+1VBE7N7BFd47XvC/'} 2024-09-13T11:25:57.937 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T11:25:57.945 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-13T11:25:57.945 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-13T11:25:57.945 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T11:25:58.022 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f899c21fbe0>, signals=[15]) 2024-09-13T11:25:58.022 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T11:25:58.031 INFO:teuthology.task.internal:Opening connections... 2024-09-13T11:25:58.032 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T11:25:58.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:25:58.117 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T11:25:58.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:25:58.201 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T11:25:58.209 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2024-09-13T11:25:58.214 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2024-09-13T11:25:58.214 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2024-09-13T11:25:58.262 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-13T11:25:58.263 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="Ubuntu" 2024-09-13T11:25:58.263 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="22.04" 2024-09-13T11:25:58.263 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-13T11:25:58.263 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_CODENAME=jammy 2024-09-13T11:25:58.263 INFO:teuthology.orchestra.run.smithi005.stdout:ID=ubuntu 2024-09-13T11:25:58.263 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE=debian 2024-09-13T11:25:58.263 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T11:25:58.264 INFO:teuthology.orchestra.run.smithi005.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T11:25:58.264 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T11:25:58.264 INFO:teuthology.orchestra.run.smithi005.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T11:25:58.264 INFO:teuthology.orchestra.run.smithi005.stdout:UBUNTU_CODENAME=jammy 2024-09-13T11:25:58.265 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2024-09-13T11:25:58.291 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-09-13T11:25:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-09-13T11:25:58.296 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-09-13T11:25:58.341 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-13T11:25:58.341 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="Ubuntu" 2024-09-13T11:25:58.342 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="22.04" 2024-09-13T11:25:58.342 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-13T11:25:58.342 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_CODENAME=jammy 2024-09-13T11:25:58.342 INFO:teuthology.orchestra.run.smithi039.stdout:ID=ubuntu 2024-09-13T11:25:58.342 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE=debian 2024-09-13T11:25:58.342 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T11:25:58.342 INFO:teuthology.orchestra.run.smithi039.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T11:25:58.342 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T11:25:58.342 INFO:teuthology.orchestra.run.smithi039.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T11:25:58.342 INFO:teuthology.orchestra.run.smithi039.stdout:UBUNTU_CODENAME=jammy 2024-09-13T11:25:58.342 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-09-13T11:25:58.366 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T11:25:58.374 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T11:25:58.395 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T11:25:58.395 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:25:58.398 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:25:58.403 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T11:25:58.419 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T11:25:58.419 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:25:58.444 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:25:58.451 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T11:25:58.508 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T11:25:58.523 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T11:25:58.524 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T11:25:58.524 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:25:58.524 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:25:58.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:25:58.529 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-09-13T11:25:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:25:58.530 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-13T11:25:58.577 INFO:teuthology.orchestra.run.smithi005.stdout:5.15.0-56-generic 2024-09-13T11:25:58.577 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.15.0-56-generic 2024-09-13T11:25:58.577 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get clean 2024-09-13T11:25:58.578 INFO:teuthology.orchestra.run.smithi039.stdout:5.15.0-56-generic 2024-09-13T11:25:58.578 INFO:teuthology.task.kernel:Running kernel on smithi039: 5.15.0-56-generic 2024-09-13T11:25:58.578 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get clean 2024-09-13T11:25:58.696 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2024-09-13T11:25:58.701 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-13T11:25:58.832 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:25:58.863 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T11:25:58.928 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-13T11:25:58.964 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T11:25:58.971 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:25:58.982 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T11:25:59.039 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T11:25:59.085 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-13T11:25:59.150 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-13T11:25:59.181 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-13T11:25:59.185 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-13T11:25:59.185 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-13T11:25:59.186 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-13T11:25:59.209 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-13T11:25:59.213 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-13T11:25:59.213 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-13T11:25:59.223 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-13T11:25:59.229 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-13T11:25:59.232 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-13T11:25:59.232 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-13T11:25:59.233 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-13T11:25:59.244 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-13T11:25:59.244 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-13T11:25:59.244 INFO:teuthology.orchestra.run.smithi005.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-13T11:25:59.244 INFO:teuthology.orchestra.run.smithi005.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-13T11:25:59.252 INFO:teuthology.orchestra.run.smithi005.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-13T11:25:59.252 INFO:teuthology.orchestra.run.smithi005.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-13T11:25:59.253 INFO:teuthology.orchestra.run.smithi005.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-13T11:25:59.253 INFO:teuthology.orchestra.run.smithi005.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-13T11:25:59.269 INFO:teuthology.orchestra.run.smithi005.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-13T11:25:59.269 INFO:teuthology.orchestra.run.smithi005.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-13T11:25:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-13T11:25:59.407 INFO:teuthology.orchestra.run.smithi005.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-13T11:25:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-13T11:25:59.533 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-13T11:25:59.693 INFO:teuthology.orchestra.run.smithi005.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-13T11:25:59.851 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-13T11:25:59.856 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-13T11:25:59.866 INFO:teuthology.orchestra.run.smithi005.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-13T11:25:59.872 INFO:teuthology.orchestra.run.smithi039.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-13T11:25:59.879 INFO:teuthology.orchestra.run.smithi005.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-13T11:25:59.880 INFO:teuthology.orchestra.run.smithi005.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-13T11:25:59.881 INFO:teuthology.orchestra.run.smithi039.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-13T11:25:59.882 INFO:teuthology.orchestra.run.smithi005.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-13T11:25:59.882 INFO:teuthology.orchestra.run.smithi039.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-13T11:25:59.887 INFO:teuthology.orchestra.run.smithi039.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-13T11:25:59.901 INFO:teuthology.orchestra.run.smithi039.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-13T11:25:59.902 INFO:teuthology.orchestra.run.smithi039.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-13T11:25:59.966 INFO:teuthology.orchestra.run.smithi005.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-13T11:25:59.972 INFO:teuthology.orchestra.run.smithi039.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-13T11:25:59.973 INFO:teuthology.orchestra.run.smithi039.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-13T11:25:59.974 INFO:teuthology.orchestra.run.smithi005.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-13T11:25:59.974 INFO:teuthology.orchestra.run.smithi005.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-13T11:25:59.977 INFO:teuthology.orchestra.run.smithi039.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-13T11:25:59.978 INFO:teuthology.orchestra.run.smithi039.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-13T11:25:59.989 INFO:teuthology.orchestra.run.smithi039.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-13T11:26:00.001 INFO:teuthology.orchestra.run.smithi005.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-13T11:26:00.026 INFO:teuthology.orchestra.run.smithi005.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-13T11:26:00.030 INFO:teuthology.orchestra.run.smithi039.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-13T11:26:00.030 INFO:teuthology.orchestra.run.smithi039.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-13T11:26:00.030 INFO:teuthology.orchestra.run.smithi039.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-13T11:26:00.031 INFO:teuthology.orchestra.run.smithi005.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-13T11:26:00.032 INFO:teuthology.orchestra.run.smithi005.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-13T11:26:00.032 INFO:teuthology.orchestra.run.smithi005.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-13T11:26:00.033 INFO:teuthology.orchestra.run.smithi005.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-13T11:26:00.046 INFO:teuthology.orchestra.run.smithi039.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-13T11:26:00.048 INFO:teuthology.orchestra.run.smithi039.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-13T11:26:00.059 INFO:teuthology.orchestra.run.smithi039.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-13T11:26:00.062 INFO:teuthology.orchestra.run.smithi039.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-13T11:26:00.063 INFO:teuthology.orchestra.run.smithi039.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-13T11:26:00.063 INFO:teuthology.orchestra.run.smithi039.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-13T11:26:00.065 INFO:teuthology.orchestra.run.smithi039.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-13T11:26:00.069 INFO:teuthology.orchestra.run.smithi039.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-13T11:26:00.070 INFO:teuthology.orchestra.run.smithi039.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-13T11:26:00.070 INFO:teuthology.orchestra.run.smithi039.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-13T11:26:00.098 INFO:teuthology.orchestra.run.smithi039.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-13T11:26:00.098 INFO:teuthology.orchestra.run.smithi039.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-13T11:26:00.111 INFO:teuthology.orchestra.run.smithi005.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-13T11:26:00.123 INFO:teuthology.orchestra.run.smithi039.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-13T11:26:00.123 INFO:teuthology.orchestra.run.smithi039.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-13T11:26:00.153 INFO:teuthology.orchestra.run.smithi039.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-13T11:26:00.154 INFO:teuthology.orchestra.run.smithi039.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-13T11:26:00.155 INFO:teuthology.orchestra.run.smithi039.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-13T11:26:00.156 INFO:teuthology.orchestra.run.smithi039.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-13T11:26:00.156 INFO:teuthology.orchestra.run.smithi039.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-13T11:26:00.179 INFO:teuthology.orchestra.run.smithi039.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-13T11:26:00.180 INFO:teuthology.orchestra.run.smithi039.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-13T11:26:00.180 INFO:teuthology.orchestra.run.smithi039.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-13T11:26:05.869 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 16.0 MB in 2s (7,245 kB/s) 2024-09-13T11:26:06.575 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 16.0 MB in 3s (5,601 kB/s) 2024-09-13T11:26:07.340 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-13T11:26:07.357 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T11:26:07.411 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-13T11:26:07.602 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2024-09-13T11:26:07.602 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2024-09-13T11:26:07.777 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:26:07.777 INFO:teuthology.orchestra.run.smithi005.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:26:07.777 INFO:teuthology.orchestra.run.smithi005.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:26:07.778 INFO:teuthology.orchestra.run.smithi005.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:26:07.778 INFO:teuthology.orchestra.run.smithi005.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-13T11:26:07.778 INFO:teuthology.orchestra.run.smithi005.stdout: python2.7-minimal 2024-09-13T11:26:07.778 INFO:teuthology.orchestra.run.smithi005.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:26:07.779 INFO:teuthology.orchestra.run.smithi005.stdout:The following additional packages will be installed: 2024-09-13T11:26:07.779 INFO:teuthology.orchestra.run.smithi005.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T11:26:07.779 INFO:teuthology.orchestra.run.smithi005.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-13T11:26:07.780 INFO:teuthology.orchestra.run.smithi005.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-13T11:26:07.781 INFO:teuthology.orchestra.run.smithi005.stdout:Suggested packages: 2024-09-13T11:26:07.781 INFO:teuthology.orchestra.run.smithi005.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-13T11:26:07.781 INFO:teuthology.orchestra.run.smithi005.stdout:Recommended packages: 2024-09-13T11:26:07.781 INFO:teuthology.orchestra.run.smithi005.stdout: thermald 2024-09-13T11:26:07.821 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2024-09-13T11:26:07.822 INFO:teuthology.orchestra.run.smithi005.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T11:26:07.822 INFO:teuthology.orchestra.run.smithi005.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-13T11:26:07.822 INFO:teuthology.orchestra.run.smithi005.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-13T11:26:07.823 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages will be upgraded: 2024-09-13T11:26:07.823 INFO:teuthology.orchestra.run.smithi005.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T11:26:08.003 INFO:teuthology.orchestra.run.smithi005.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T11:26:08.003 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 113 MB of archives. 2024-09-13T11:26:08.003 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-13T11:26:08.003 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-13T11:26:08.045 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-13T11:26:08.065 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T11:26:08.132 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-13T11:26:08.348 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-13T11:26:08.348 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-13T11:26:08.544 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:26:08.545 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:26:08.545 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:26:08.545 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:26:08.545 INFO:teuthology.orchestra.run.smithi039.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-13T11:26:08.545 INFO:teuthology.orchestra.run.smithi039.stdout: python2.7-minimal 2024-09-13T11:26:08.546 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:26:08.547 INFO:teuthology.orchestra.run.smithi039.stdout:The following additional packages will be installed: 2024-09-13T11:26:08.547 INFO:teuthology.orchestra.run.smithi039.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T11:26:08.547 INFO:teuthology.orchestra.run.smithi039.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-13T11:26:08.547 INFO:teuthology.orchestra.run.smithi039.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-13T11:26:08.549 INFO:teuthology.orchestra.run.smithi039.stdout:Suggested packages: 2024-09-13T11:26:08.549 INFO:teuthology.orchestra.run.smithi039.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-13T11:26:08.549 INFO:teuthology.orchestra.run.smithi039.stdout:Recommended packages: 2024-09-13T11:26:08.549 INFO:teuthology.orchestra.run.smithi039.stdout: thermald 2024-09-13T11:26:08.604 INFO:teuthology.orchestra.run.smithi039.stdout:The following NEW packages will be installed: 2024-09-13T11:26:08.604 INFO:teuthology.orchestra.run.smithi039.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T11:26:08.604 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-13T11:26:08.604 INFO:teuthology.orchestra.run.smithi039.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-13T11:26:08.605 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages will be upgraded: 2024-09-13T11:26:08.606 INFO:teuthology.orchestra.run.smithi039.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T11:26:08.658 INFO:teuthology.orchestra.run.smithi039.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T11:26:08.658 INFO:teuthology.orchestra.run.smithi039.stdout:Need to get 113 MB of archives. 2024-09-13T11:26:08.658 INFO:teuthology.orchestra.run.smithi039.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-13T11:26:08.658 INFO:teuthology.orchestra.run.smithi039.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-13T11:26:09.111 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-13T11:26:09.210 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-13T11:26:09.421 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-13T11:26:09.772 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-13T11:26:10.050 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-13T11:26:10.050 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-13T11:26:10.050 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-13T11:26:10.178 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-13T11:26:10.356 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-13T11:26:10.800 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 113 MB in 2s (65.0 MB/s) 2024-09-13T11:26:10.980 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-13T11:26:12.022 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-13T11:26:12.022 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-13T11:26:12.022 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-13T11:26:12.460 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-13T11:26:12.517 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-13T11:26:12.926 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 113 MB in 5s (24.2 MB/s) 2024-09-13T11:26:12.945 INFO:teuthology.orchestra.run.smithi039.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-13T11:26:12.950 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:26:13.010 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:13.093 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-13T11:26:15.039 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-13T11:26:15.043 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:26:15.107 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:15.460 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-13T11:26:15.482 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:26:15.605 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:16.005 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-13T11:26:16.026 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:26:16.063 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:17.392 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-13T11:26:17.414 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:26:17.511 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:17.945 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-13T11:26:17.965 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:26:18.003 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:22.745 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:26:22.854 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:26:23.171 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:26:23.317 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:26:23.628 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-13T11:26:23.650 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-13T11:26:23.694 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T11:26:23.704 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:26:23.806 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:26:24.131 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:26:24.259 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:26:24.563 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-13T11:26:24.586 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-13T11:26:24.638 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T11:26:29.474 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-13T11:26:29.511 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:26:29.549 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:30.634 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-13T11:26:30.657 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T11:26:30.709 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:32.234 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:26:32.362 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:26:32.687 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T11:26:32.817 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:32.943 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-13T11:26:33.071 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:33.338 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T11:26:33.446 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T11:26:33.745 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T11:26:33.867 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:33.994 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-13T11:26:34.121 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:34.618 INFO:teuthology.orchestra.run.smithi039.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:26:34.619 INFO:teuthology.orchestra.run.smithi039.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-13T11:26:34.619 INFO:teuthology.orchestra.run.smithi039.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:26:34.619 INFO:teuthology.orchestra.run.smithi039.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-13T11:26:35.242 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:35.493 INFO:teuthology.orchestra.run.smithi005.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:26:35.494 INFO:teuthology.orchestra.run.smithi005.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-13T11:26:35.494 INFO:teuthology.orchestra.run.smithi005.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:26:35.494 INFO:teuthology.orchestra.run.smithi005.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-13T11:26:35.665 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:36.668 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:37.101 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:38.078 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-13T11:26:38.203 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-13T11:26:38.330 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:38.400 INFO:teuthology.orchestra.run.smithi039.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T11:26:38.400 INFO:teuthology.orchestra.run.smithi039.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:26:38.527 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-13T11:26:38.653 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-13T11:26:38.779 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T11:26:38.844 INFO:teuthology.orchestra.run.smithi005.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T11:26:38.844 INFO:teuthology.orchestra.run.smithi005.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:26:52.837 INFO:teuthology.orchestra.run.smithi039.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T11:26:52.837 INFO:teuthology.orchestra.run.smithi039.stdout:Sourcing file `/etc/default/grub' 2024-09-13T11:26:52.848 INFO:teuthology.orchestra.run.smithi039.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T11:26:52.874 INFO:teuthology.orchestra.run.smithi039.stdout:Generating grub configuration file ... 2024-09-13T11:26:53.106 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:26:53.124 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi005.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T11:26:53.377 INFO:teuthology.orchestra.run.smithi005.stdout:Sourcing file `/etc/default/grub' 2024-09-13T11:26:53.387 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:26:53.393 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T11:26:53.393 INFO:teuthology.orchestra.run.smithi005.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T11:26:53.412 INFO:teuthology.orchestra.run.smithi005.stdout:Generating grub configuration file ... 2024-09-13T11:26:53.465 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T11:26:53.470 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi039.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi039.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T11:26:53.611 INFO:teuthology.orchestra.run.smithi039.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T11:26:53.660 INFO:teuthology.orchestra.run.smithi005.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:26:53.666 INFO:teuthology.orchestra.run.smithi039.stdout:done 2024-09-13T11:26:53.679 INFO:teuthology.orchestra.run.smithi005.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:26:54.038 INFO:teuthology.orchestra.run.smithi005.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:26:54.044 INFO:teuthology.orchestra.run.smithi005.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T11:26:54.125 INFO:teuthology.orchestra.run.smithi005.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T11:26:54.131 INFO:teuthology.orchestra.run.smithi005.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T11:26:54.186 DEBUG:teuthology.orchestra.run.smithi039:> dpkg -s linux-image-generic 2024-09-13T11:26:54.216 INFO:teuthology.orchestra.run.smithi039.stdout:Package: linux-image-generic 2024-09-13T11:26:54.216 INFO:teuthology.orchestra.run.smithi039.stdout:Status: install ok installed 2024-09-13T11:26:54.216 INFO:teuthology.orchestra.run.smithi039.stdout:Priority: optional 2024-09-13T11:26:54.216 INFO:teuthology.orchestra.run.smithi039.stdout:Section: kernel 2024-09-13T11:26:54.216 INFO:teuthology.orchestra.run.smithi039.stdout:Installed-Size: 21 2024-09-13T11:26:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T11:26:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: amd64 2024-09-13T11:26:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:Source: linux-meta 2024-09-13T11:26:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:Version: 5.15.0.121.121 2024-09-13T11:26:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T11:26:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T11:26:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:Recommends: thermald 2024-09-13T11:26:54.217 INFO:teuthology.orchestra.run.smithi039.stdout:Description: Generic Linux kernel image 2024-09-13T11:26:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: This package will always depend on the latest generic kernel image 2024-09-13T11:26:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: available. 2024-09-13T11:26:54.218 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-13T11:26:54.218 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:26:54.218 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:26:54.218 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T11:26:54.219 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:26:54.219 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get clean 2024-09-13T11:26:54.297 INFO:teuthology.orchestra.run.smithi005.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T11:26:54.298 INFO:teuthology.orchestra.run.smithi005.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T11:26:54.298 INFO:teuthology.orchestra.run.smithi005.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T11:26:54.336 INFO:teuthology.orchestra.run.smithi005.stdout:done 2024-09-13T11:26:54.419 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-13T11:26:54.535 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T11:26:54.674 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:26:54.752 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T11:26:54.799 DEBUG:teuthology.orchestra.run.smithi005:> dpkg -s linux-image-generic 2024-09-13T11:26:54.818 INFO:teuthology.orchestra.run.smithi005.stdout:Package: linux-image-generic 2024-09-13T11:26:54.818 INFO:teuthology.orchestra.run.smithi005.stdout:Status: install ok installed 2024-09-13T11:26:54.818 INFO:teuthology.orchestra.run.smithi005.stdout:Priority: optional 2024-09-13T11:26:54.818 INFO:teuthology.orchestra.run.smithi005.stdout:Section: kernel 2024-09-13T11:26:54.818 INFO:teuthology.orchestra.run.smithi005.stdout:Installed-Size: 21 2024-09-13T11:26:54.818 INFO:teuthology.orchestra.run.smithi005.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T11:26:54.819 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: amd64 2024-09-13T11:26:54.819 INFO:teuthology.orchestra.run.smithi005.stdout:Source: linux-meta 2024-09-13T11:26:54.819 INFO:teuthology.orchestra.run.smithi005.stdout:Version: 5.15.0.121.121 2024-09-13T11:26:54.819 INFO:teuthology.orchestra.run.smithi005.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T11:26:54.819 INFO:teuthology.orchestra.run.smithi005.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T11:26:54.819 INFO:teuthology.orchestra.run.smithi005.stdout:Recommends: thermald 2024-09-13T11:26:54.819 INFO:teuthology.orchestra.run.smithi005.stdout:Description: Generic Linux kernel image 2024-09-13T11:26:54.819 INFO:teuthology.orchestra.run.smithi005.stdout: This package will always depend on the latest generic kernel image 2024-09-13T11:26:54.819 INFO:teuthology.orchestra.run.smithi005.stdout: available. 2024-09-13T11:26:54.819 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-13T11:26:54.819 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:26:54.819 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:26:54.820 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T11:26:54.820 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:26:54.820 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get clean 2024-09-13T11:26:54.977 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2024-09-13T11:26:55.099 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:26:55.099 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T11:26:55.107 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T11:26:55.108 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T11:26:55.124 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T11:26:55.308 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 128 kB in 1s (176 kB/s) 2024-09-13T11:26:56.374 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-13T11:26:56.393 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T11:26:56.449 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-13T11:26:56.545 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-13T11:26:56.562 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T11:26:56.625 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-13T11:26:56.658 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-13T11:26:56.658 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-13T11:26:56.837 INFO:teuthology.orchestra.run.smithi039.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-13T11:26:56.840 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:26:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:26:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:26:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:26:56.841 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T11:26:56.841 INFO:teuthology.orchestra.run.smithi039.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T11:26:56.841 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:26:56.857 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2024-09-13T11:26:56.859 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2024-09-13T11:26:56.881 INFO:teuthology.orchestra.run.smithi039.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T11:26:56.883 DEBUG:teuthology.orchestra.run.smithi039:> dpkg -s linux-image-generic 2024-09-13T11:26:56.899 INFO:teuthology.orchestra.run.smithi039.stdout:Package: linux-image-generic 2024-09-13T11:26:56.899 INFO:teuthology.orchestra.run.smithi039.stdout:Status: install ok installed 2024-09-13T11:26:56.899 INFO:teuthology.orchestra.run.smithi039.stdout:Priority: optional 2024-09-13T11:26:56.899 INFO:teuthology.orchestra.run.smithi039.stdout:Section: kernel 2024-09-13T11:26:56.899 INFO:teuthology.orchestra.run.smithi039.stdout:Installed-Size: 21 2024-09-13T11:26:56.899 INFO:teuthology.orchestra.run.smithi039.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T11:26:56.899 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: amd64 2024-09-13T11:26:56.899 INFO:teuthology.orchestra.run.smithi039.stdout:Source: linux-meta 2024-09-13T11:26:56.900 INFO:teuthology.orchestra.run.smithi039.stdout:Version: 5.15.0.121.121 2024-09-13T11:26:56.900 INFO:teuthology.orchestra.run.smithi039.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T11:26:56.900 INFO:teuthology.orchestra.run.smithi039.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T11:26:56.900 INFO:teuthology.orchestra.run.smithi039.stdout:Recommends: thermald 2024-09-13T11:26:56.900 INFO:teuthology.orchestra.run.smithi039.stdout:Description: Generic Linux kernel image 2024-09-13T11:26:56.900 INFO:teuthology.orchestra.run.smithi039.stdout: This package will always depend on the latest generic kernel image 2024-09-13T11:26:56.900 INFO:teuthology.orchestra.run.smithi039.stdout: available. 2024-09-13T11:26:56.900 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-13T11:26:56.944 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.3yACkgv1Cy 2024-09-13T11:26:56.945 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub/grub.cfg /tmp/tmp.3yACkgv1Cy 2024-09-13T11:26:57.000 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.3yACkgv1Cy 2024-09-13T11:26:57.069 INFO:teuthology.orchestra.run.smithi005.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-13T11:26:57.069 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:26:57.069 INFO:teuthology.orchestra.run.smithi005.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:26:57.069 INFO:teuthology.orchestra.run.smithi005.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:26:57.069 INFO:teuthology.orchestra.run.smithi005.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:26:57.070 INFO:teuthology.orchestra.run.smithi005.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T11:26:57.070 INFO:teuthology.orchestra.run.smithi005.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T11:26:57.070 INFO:teuthology.orchestra.run.smithi005.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:26:57.116 INFO:teuthology.orchestra.run.smithi005.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T11:26:57.118 DEBUG:teuthology.orchestra.run.smithi005:> dpkg -s linux-image-generic 2024-09-13T11:26:57.131 INFO:teuthology.orchestra.run.smithi005.stdout:Package: linux-image-generic 2024-09-13T11:26:57.131 INFO:teuthology.orchestra.run.smithi005.stdout:Status: install ok installed 2024-09-13T11:26:57.131 INFO:teuthology.orchestra.run.smithi005.stdout:Priority: optional 2024-09-13T11:26:57.131 INFO:teuthology.orchestra.run.smithi005.stdout:Section: kernel 2024-09-13T11:26:57.131 INFO:teuthology.orchestra.run.smithi005.stdout:Installed-Size: 21 2024-09-13T11:26:57.131 INFO:teuthology.orchestra.run.smithi005.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T11:26:57.131 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: amd64 2024-09-13T11:26:57.131 INFO:teuthology.orchestra.run.smithi005.stdout:Source: linux-meta 2024-09-13T11:26:57.132 INFO:teuthology.orchestra.run.smithi005.stdout:Version: 5.15.0.121.121 2024-09-13T11:26:57.132 INFO:teuthology.orchestra.run.smithi005.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T11:26:57.132 INFO:teuthology.orchestra.run.smithi005.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T11:26:57.132 INFO:teuthology.orchestra.run.smithi005.stdout:Recommends: thermald 2024-09-13T11:26:57.132 INFO:teuthology.orchestra.run.smithi005.stdout:Description: Generic Linux kernel image 2024-09-13T11:26:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: This package will always depend on the latest generic kernel image 2024-09-13T11:26:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: available. 2024-09-13T11:26:57.132 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-09-13T11:26:57.140 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.3yACkgv1Cy is 10KB 2024-09-13T11:26:57.152 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.3yACkgv1Cy 2024-09-13T11:26:57.158 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T11:26:57.177 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.VZMZQUEaEL 2024-09-13T11:26:57.178 DEBUG:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub/grub.cfg /tmp/tmp.VZMZQUEaEL 2024-09-13T11:26:57.212 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:26:57.212 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T11:26:57.212 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T11:26:57.234 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.VZMZQUEaEL 2024-09-13T11:26:57.283 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-13T11:26:57.283 DEBUG:teuthology.orchestra.run.smithi039:> sudo update-grub 2024-09-13T11:26:57.354 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.VZMZQUEaEL is 10KB 2024-09-13T11:26:57.367 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.VZMZQUEaEL 2024-09-13T11:26:57.372 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T11:26:57.427 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:26:57.427 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T11:26:57.427 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T11:26:57.501 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-13T11:26:57.501 DEBUG:teuthology.orchestra.run.smithi005:> sudo update-grub 2024-09-13T11:26:58.433 INFO:teuthology.orchestra.run.smithi039.stderr:Sourcing file `/etc/default/grub' 2024-09-13T11:26:58.434 INFO:teuthology.orchestra.run.smithi039.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T11:26:58.437 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-09-13T11:26:58.674 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:26:58.680 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:26:58.689 INFO:teuthology.orchestra.run.smithi005.stderr:Sourcing file `/etc/default/grub' 2024-09-13T11:26:58.690 INFO:teuthology.orchestra.run.smithi005.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T11:26:58.692 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2024-09-13T11:26:58.883 INFO:teuthology.orchestra.run.smithi005.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T11:26:58.890 INFO:teuthology.orchestra.run.smithi005.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T11:26:58.900 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:26:58.905 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T11:26:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T11:26:58.948 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T11:26:59.099 INFO:teuthology.orchestra.run.smithi039.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T11:26:59.099 INFO:teuthology.orchestra.run.smithi039.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T11:26:59.099 INFO:teuthology.orchestra.run.smithi039.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T11:26:59.119 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-09-13T11:26:59.120 INFO:teuthology.orchestra.run.smithi005.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T11:26:59.121 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-09-13T11:26:59.125 INFO:teuthology.orchestra.run.smithi005.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T11:26:59.172 INFO:teuthology.orchestra.run.smithi005.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T11:26:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T11:26:59.339 INFO:teuthology.orchestra.run.smithi005.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T11:26:59.339 INFO:teuthology.orchestra.run.smithi005.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T11:26:59.339 INFO:teuthology.orchestra.run.smithi005.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T11:26:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:done 2024-09-13T11:26:59.361 DEBUG:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2024-09-13T11:27:29.153 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T11:27:29.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:27:29.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:27:29.364 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T11:27:29.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T11:27:29.365 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:28:03.645 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-13T11:28:12.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:28:12.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:28:15.713 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-13T11:28:27.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:28:27.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:28:29.369 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T11:28:30.781 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-13T11:28:38.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T11:28:38.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:28:45.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:28:45.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:28:45.986 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-09-13T11:28:46.633 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T11:28:46.634 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-13T11:28:46.634 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-09-13T11:28:46.680 INFO:teuthology.orchestra.run.smithi005.stdout:5.15.0-121-generic 2024-09-13T11:28:46.680 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-13T11:28:46.680 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:28:46.681 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T11:28:47.682 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T11:28:47.682 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:28:47.768 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2024-09-13T11:28:47.802 DEBUG:teuthology.parallel:result is None 2024-09-13T11:29:19.354 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-13T11:29:37.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:29:37.357 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:29:37.563 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-13T11:29:38.338 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:29:38.339 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-13T11:29:38.339 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-13T11:29:38.387 INFO:teuthology.orchestra.run.smithi039.stdout:5.15.0-121-generic 2024-09-13T11:29:38.388 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-13T11:29:38.388 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:29:38.388 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T11:29:39.389 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T11:29:39.389 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:29:39.457 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2024-09-13T11:29:39.477 DEBUG:teuthology.parallel:result is None 2024-09-13T11:29:39.477 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T11:29:39.484 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T11:29:39.484 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:29:39.487 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:29:39.493 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T11:29:39.531 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T11:29:39.565 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T11:29:39.565 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:29:39.568 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:29:39.586 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T11:29:39.604 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T11:29:39.604 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:29:39.620 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:29:39.636 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:29:39.644 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:29:39.646 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:29:39.654 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:29:39.656 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T11:29:39.663 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T11:29:39.664 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:29:39.692 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:29:39.715 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T11:29:39.726 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T11:29:39.726 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:29:39.760 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:29:39.765 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:29:39.811 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:29:39.859 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:29:39.859 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:29:39.918 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:29:39.924 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:29:39.972 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:29:39.972 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:29:40.056 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2024-09-13T11:29:40.058 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-09-13T11:29:40.140 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T11:29:40.148 INFO:teuthology.task.internal:Starting timer... 2024-09-13T11:29:40.148 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T11:29:40.202 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T11:29:40.233 DEBUG:teuthology.task.selinux:Excluding smithi005: OS 'ubuntu' does not support SELinux 2024-09-13T11:29:40.233 DEBUG:teuthology.task.selinux:Excluding smithi039: OS 'ubuntu' does not support SELinux 2024-09-13T11:29:40.233 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T11:29:40.234 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-13T11:29:40.234 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T11:29:40.234 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T11:29:40.267 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-13T11:29:40.271 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-13T11:29:40.381 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T11:29:40.392 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T11:29:40.393 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi039.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T11:37:21.457 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi039.front.sepia.ceph.com')] 2024-09-13T11:37:21.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T11:37:21.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:37:21.571 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-09-13T11:37:21.653 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T11:37:21.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:37:21.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:37:21.761 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-13T11:37:21.837 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:37:21.838 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T11:37:21.847 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T11:37:21.848 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:37:21.848 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:37:21.851 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:37:21.851 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:37:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-13T11:37:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: Command line: ntpd -gq 2024-09-13T11:37:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: ---------------------------------------------------- 2024-09-13T11:37:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: ntp-4 is maintained by Network Time Foundation, 2024-09-13T11:37:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-13T11:37:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: corporation. Support and training for ntp-4 are 2024-09-13T11:37:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: available at https://www.nwtime.org/support 2024-09-13T11:37:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: ---------------------------------------------------- 2024-09-13T11:37:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: proto: precision = 0.061 usec (-24) 2024-09-13T11:37:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: basedate set to 2022-02-04 2024-09-13T11:37:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: gps base set to 2022-02-06 (week 2196) 2024-09-13T11:37:21.882 INFO:teuthology.orchestra.run.smithi005.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T11:37:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T11:37:21.883 INFO:teuthology.orchestra.run.smithi005.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T11:37:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T11:37:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T11:37:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T11:37:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T11:37:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: Listen normally on 3 ens1f0 172.21.15.5:123 2024-09-13T11:37:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: Listen normally on 4 lo [::1]:123 2024-09-13T11:37:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1442%3]:123 2024-09-13T11:37:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:21 ntpd[18409]: Listening on routing socket on fd #22 for interface updates 2024-09-13T11:37:21.913 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-13T11:37:21.914 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: Command line: ntpd -gq 2024-09-13T11:37:21.914 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: ---------------------------------------------------- 2024-09-13T11:37:21.914 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: ntp-4 is maintained by Network Time Foundation, 2024-09-13T11:37:21.914 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-13T11:37:21.914 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: corporation. Support and training for ntp-4 are 2024-09-13T11:37:21.914 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: available at https://www.nwtime.org/support 2024-09-13T11:37:21.914 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: ---------------------------------------------------- 2024-09-13T11:37:21.914 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: proto: precision = 0.044 usec (-24) 2024-09-13T11:37:21.914 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: basedate set to 2022-02-04 2024-09-13T11:37:21.914 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: gps base set to 2022-02-06 (week 2196) 2024-09-13T11:37:21.915 INFO:teuthology.orchestra.run.smithi039.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T11:37:21.915 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T11:37:21.915 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T11:37:21.915 INFO:teuthology.orchestra.run.smithi039.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T11:37:21.916 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T11:37:21.916 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T11:37:21.917 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T11:37:21.917 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: Listen normally on 3 ens1f0 172.21.15.39:123 2024-09-13T11:37:21.917 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: Listen normally on 4 lo [::1]:123 2024-09-13T11:37:21.917 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1370%4]:123 2024-09-13T11:37:21.917 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:21 ntpd[18356]: Listening on routing socket on fd #22 for interface updates 2024-09-13T11:37:22.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:22 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:22.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:22 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:22.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:22 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:22.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:22 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:22.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:22 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:22.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:22 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:22.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:22 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:22.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:22 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:22.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:22 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:22.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:22 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:22.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:22 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:22.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:22 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:22.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:22 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:22.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:22 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:23.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:23 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:23.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:23 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:23.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:23 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:23.916 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:23 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:23.916 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:23 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:23.916 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:23 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:23.916 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:23 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:23.916 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:23 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:23.916 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:23 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:23.916 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:23 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:23.916 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:23 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:23.917 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:23 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:24.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:24 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:24.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:24 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:24.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:24 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:24.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:24 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:24.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:24 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:24.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:24 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:24.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:24 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:24.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:24 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:24.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:24 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:24.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:24 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:24.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:24 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:24.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:24 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:24.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:24 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:24.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:24 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:24.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:24 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:24.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:24 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:24.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:24 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:24.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:24 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:25.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:25 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:25 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:25 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:25 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:25 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:25 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:25 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:25 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:25 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:25 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:25 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:25 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:26.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:26.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:26.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:26.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:26.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:26.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:26.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:26.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:26.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:26.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:26.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:26.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:26.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:26.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:26.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:26.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:26.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:26.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:26 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:26.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:26 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:26.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:26 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:26.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:26 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:26.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:26 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:26.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:26 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:26.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:26 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:26.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:26 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:26.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:26 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:27.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:27 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:27.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:27 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:27.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:27 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:27.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:27 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:27.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:27 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:27 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:27 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:27 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:27 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:27 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:27 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:27 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:28.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:28 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:28.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:28 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:28.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:28 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:28.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:28 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:28.881 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:28 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:28.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:28 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:28.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:28 ntpd[18409]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:28.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:28 ntpd[18409]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:28.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:28 ntpd[18409]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:28.882 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:28 ntpd[18409]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:28.882 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 11:37:28 ntpd[18409]: ntpd: time slew -0.000697 s 2024-09-13T11:37:28.882 INFO:teuthology.orchestra.run.smithi005.stdout:ntpd: time slew -0.000697s 2024-09-13T11:37:28.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:28 ntpd[18409]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T11:37:28.883 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 11:37:28 ntpd[18409]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T11:37:28.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:28 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:28.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:28 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:28.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:28 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T11:37:28.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:28 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T11:37:28.914 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:28 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T11:37:28.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:28 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T11:37:28.915 INFO:teuthology.orchestra.run.smithi039.stdout:13 Sep 11:37:28 ntpd[18356]: ntpd: time slew +0.001511 s 2024-09-13T11:37:28.915 INFO:teuthology.orchestra.run.smithi039.stdout:ntpd: time slew +0.001511s 2024-09-13T11:37:28.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:28 ntpd[18356]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T11:37:28.915 INFO:teuthology.orchestra.run.smithi039.stderr:13 Sep 11:37:28 ntpd[18356]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T11:37:28.926 INFO:teuthology.orchestra.run.smithi005.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T11:37:28.926 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================== 2024-09-13T11:37:28.926 INFO:teuthology.orchestra.run.smithi005.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:37:28.926 INFO:teuthology.orchestra.run.smithi005.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:37:28.926 INFO:teuthology.orchestra.run.smithi005.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:37:28.926 INFO:teuthology.orchestra.run.smithi005.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:37:28.969 INFO:teuthology.orchestra.run.smithi039.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T11:37:28.969 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================== 2024-09-13T11:37:28.970 INFO:teuthology.orchestra.run.smithi039.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:37:28.970 INFO:teuthology.orchestra.run.smithi039.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:37:28.970 INFO:teuthology.orchestra.run.smithi039.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:37:28.970 INFO:teuthology.orchestra.run.smithi039.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:37:28.970 INFO:teuthology.run_tasks:Running task install... 2024-09-13T11:37:29.126 DEBUG:teuthology.task.install:project ceph 2024-09-13T11:37:29.126 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T11:37:29.126 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T11:37:29.126 INFO:teuthology.task.install:Using flavor: default 2024-09-13T11:37:29.132 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T11:37:29.132 INFO:teuthology.task.install:extra packages: [] 2024-09-13T11:37:29.133 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-key list | grep Ceph 2024-09-13T11:37:29.133 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-key list | grep Ceph 2024-09-13T11:37:29.199 INFO:teuthology.orchestra.run.smithi005.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-13T11:37:29.200 INFO:teuthology.orchestra.run.smithi039.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-13T11:37:29.231 INFO:teuthology.orchestra.run.smithi005.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T11:37:29.231 INFO:teuthology.orchestra.run.smithi005.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T11:37:29.231 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-13T11:37:29.232 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:37:29.244 INFO:teuthology.orchestra.run.smithi039.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T11:37:29.244 INFO:teuthology.orchestra.run.smithi039.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T11:37:29.245 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-13T11:37:29.245 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:37:29.387 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default/ 2024-09-13T11:37:29.387 INFO:teuthology.task.install.deb:Package version is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:37:29.402 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default/ 2024-09-13T11:37:29.402 INFO:teuthology.task.install.deb:Package version is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:37:29.510 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:37:29.510 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T11:37:29.519 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:37:29.520 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T11:37:29.524 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2024-09-13T11:37:29.534 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-13T11:37:29.700 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:37:29.700 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T11:37:29.710 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T11:37:29.727 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T11:37:29.751 INFO:teuthology.orchestra.run.smithi005.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy InRelease 2024-09-13T11:37:29.754 INFO:teuthology.orchestra.run.smithi039.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy InRelease 2024-09-13T11:37:29.772 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-13T11:37:29.775 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-13T11:37:29.796 INFO:teuthology.orchestra.run.smithi005.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-13T11:37:29.801 INFO:teuthology.orchestra.run.smithi039.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-13T11:37:29.822 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-13T11:37:29.827 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T11:37:29.827 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T11:37:29.827 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,426 B] 2024-09-13T11:37:29.850 INFO:teuthology.orchestra.run.smithi039.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,426 B] 2024-09-13T11:37:29.856 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-13T11:37:29.898 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T11:37:29.980 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T11:37:30.075 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 29.3 kB in 0s (82.4 kB/s) 2024-09-13T11:37:30.217 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 29.3 kB in 1s (58.2 kB/s) 2024-09-13T11:37:31.134 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-13T11:37:31.152 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4697-gd58f7306-1jammy cephadm=19.3.0-4697-gd58f7306-1jammy ceph-mds=19.3.0-4697-gd58f7306-1jammy ceph-mgr=19.3.0-4697-gd58f7306-1jammy ceph-common=19.3.0-4697-gd58f7306-1jammy ceph-fuse=19.3.0-4697-gd58f7306-1jammy ceph-test=19.3.0-4697-gd58f7306-1jammy ceph-volume=19.3.0-4697-gd58f7306-1jammy radosgw=19.3.0-4697-gd58f7306-1jammy python3-rados=19.3.0-4697-gd58f7306-1jammy python3-rgw=19.3.0-4697-gd58f7306-1jammy python3-cephfs=19.3.0-4697-gd58f7306-1jammy python3-rbd=19.3.0-4697-gd58f7306-1jammy libcephfs2=19.3.0-4697-gd58f7306-1jammy libcephfs-dev=19.3.0-4697-gd58f7306-1jammy librados2=19.3.0-4697-gd58f7306-1jammy librbd1=19.3.0-4697-gd58f7306-1jammy rbd-fuse=19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:37:31.213 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-13T11:37:31.272 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-13T11:37:31.289 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4697-gd58f7306-1jammy cephadm=19.3.0-4697-gd58f7306-1jammy ceph-mds=19.3.0-4697-gd58f7306-1jammy ceph-mgr=19.3.0-4697-gd58f7306-1jammy ceph-common=19.3.0-4697-gd58f7306-1jammy ceph-fuse=19.3.0-4697-gd58f7306-1jammy ceph-test=19.3.0-4697-gd58f7306-1jammy ceph-volume=19.3.0-4697-gd58f7306-1jammy radosgw=19.3.0-4697-gd58f7306-1jammy python3-rados=19.3.0-4697-gd58f7306-1jammy python3-rgw=19.3.0-4697-gd58f7306-1jammy python3-cephfs=19.3.0-4697-gd58f7306-1jammy python3-rbd=19.3.0-4697-gd58f7306-1jammy libcephfs2=19.3.0-4697-gd58f7306-1jammy libcephfs-dev=19.3.0-4697-gd58f7306-1jammy librados2=19.3.0-4697-gd58f7306-1jammy librbd1=19.3.0-4697-gd58f7306-1jammy rbd-fuse=19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:37:31.362 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-13T11:37:31.420 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-13T11:37:31.421 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-13T11:37:31.571 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2024-09-13T11:37:31.572 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2024-09-13T11:37:31.639 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:37:31.639 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:37:31.640 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:37:31.640 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:37:31.640 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T11:37:31.640 INFO:teuthology.orchestra.run.smithi039.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T11:37:31.641 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:37:31.642 INFO:teuthology.orchestra.run.smithi039.stdout:The following additional packages will be installed: 2024-09-13T11:37:31.642 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T11:37:31.642 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-13T11:37:31.642 INFO:teuthology.orchestra.run.smithi039.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-13T11:37:31.642 INFO:teuthology.orchestra.run.smithi039.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-13T11:37:31.643 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-13T11:37:31.643 INFO:teuthology.orchestra.run.smithi039.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-13T11:37:31.643 INFO:teuthology.orchestra.run.smithi039.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-13T11:37:31.643 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-13T11:37:31.643 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-13T11:37:31.644 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T11:37:31.644 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T11:37:31.644 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T11:37:31.644 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T11:37:31.644 INFO:teuthology.orchestra.run.smithi039.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T11:37:31.644 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T11:37:31.644 INFO:teuthology.orchestra.run.smithi039.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T11:37:31.644 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-13T11:37:31.644 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T11:37:31.644 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T11:37:31.645 INFO:teuthology.orchestra.run.smithi039.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T11:37:31.645 INFO:teuthology.orchestra.run.smithi039.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T11:37:31.645 INFO:teuthology.orchestra.run.smithi039.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T11:37:31.645 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-13T11:37:31.645 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet zip 2024-09-13T11:37:31.645 INFO:teuthology.orchestra.run.smithi039.stdout:Suggested packages: 2024-09-13T11:37:31.645 INFO:teuthology.orchestra.run.smithi039.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-13T11:37:31.645 INFO:teuthology.orchestra.run.smithi039.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-13T11:37:31.645 INFO:teuthology.orchestra.run.smithi039.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-13T11:37:31.645 INFO:teuthology.orchestra.run.smithi039.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-13T11:37:31.646 INFO:teuthology.orchestra.run.smithi039.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-13T11:37:31.646 INFO:teuthology.orchestra.run.smithi039.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-13T11:37:31.646 INFO:teuthology.orchestra.run.smithi039.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-13T11:37:31.646 INFO:teuthology.orchestra.run.smithi039.stdout:Recommended packages: 2024-09-13T11:37:31.646 INFO:teuthology.orchestra.run.smithi039.stdout: btrfs-tools 2024-09-13T11:37:31.757 INFO:teuthology.orchestra.run.smithi039.stdout:The following NEW packages will be installed: 2024-09-13T11:37:31.757 INFO:teuthology.orchestra.run.smithi039.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T11:37:31.757 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T11:37:31.757 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-13T11:37:31.758 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-13T11:37:31.758 INFO:teuthology.orchestra.run.smithi039.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-13T11:37:31.758 INFO:teuthology.orchestra.run.smithi039.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-13T11:37:31.759 INFO:teuthology.orchestra.run.smithi039.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-13T11:37:31.759 INFO:teuthology.orchestra.run.smithi039.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-13T11:37:31.759 INFO:teuthology.orchestra.run.smithi039.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-13T11:37:31.759 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-13T11:37:31.759 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-13T11:37:31.760 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T11:37:31.760 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T11:37:31.760 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T11:37:31.760 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T11:37:31.760 INFO:teuthology.orchestra.run.smithi039.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T11:37:31.760 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T11:37:31.760 INFO:teuthology.orchestra.run.smithi039.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T11:37:31.760 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-13T11:37:31.760 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-13T11:37:31.760 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-13T11:37:31.761 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-13T11:37:31.761 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-13T11:37:31.761 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-13T11:37:31.761 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-13T11:37:31.761 INFO:teuthology.orchestra.run.smithi039.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-13T11:37:31.786 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T11:37:31.786 INFO:teuthology.orchestra.run.smithi005.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T11:37:31.786 INFO:teuthology.orchestra.run.smithi005.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T11:37:31.787 INFO:teuthology.orchestra.run.smithi005.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T11:37:31.787 INFO:teuthology.orchestra.run.smithi005.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T11:37:31.787 INFO:teuthology.orchestra.run.smithi005.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T11:37:31.788 INFO:teuthology.orchestra.run.smithi005.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T11:37:31.788 INFO:teuthology.orchestra.run.smithi005.stdout:The following additional packages will be installed: 2024-09-13T11:37:31.788 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T11:37:31.789 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-13T11:37:31.789 INFO:teuthology.orchestra.run.smithi005.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-13T11:37:31.789 INFO:teuthology.orchestra.run.smithi005.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-13T11:37:31.789 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-13T11:37:31.790 INFO:teuthology.orchestra.run.smithi005.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-13T11:37:31.790 INFO:teuthology.orchestra.run.smithi005.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-13T11:37:31.790 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-13T11:37:31.790 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-13T11:37:31.791 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T11:37:31.791 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T11:37:31.791 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T11:37:31.791 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T11:37:31.792 INFO:teuthology.orchestra.run.smithi005.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T11:37:31.792 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T11:37:31.792 INFO:teuthology.orchestra.run.smithi005.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T11:37:31.792 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-13T11:37:31.793 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T11:37:31.793 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T11:37:31.793 INFO:teuthology.orchestra.run.smithi005.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T11:37:31.793 INFO:teuthology.orchestra.run.smithi005.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T11:37:31.793 INFO:teuthology.orchestra.run.smithi005.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T11:37:31.793 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-13T11:37:31.793 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet zip 2024-09-13T11:37:31.794 INFO:teuthology.orchestra.run.smithi005.stdout:Suggested packages: 2024-09-13T11:37:31.794 INFO:teuthology.orchestra.run.smithi005.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-13T11:37:31.794 INFO:teuthology.orchestra.run.smithi005.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-13T11:37:31.794 INFO:teuthology.orchestra.run.smithi005.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-13T11:37:31.794 INFO:teuthology.orchestra.run.smithi005.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-13T11:37:31.794 INFO:teuthology.orchestra.run.smithi005.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-13T11:37:31.794 INFO:teuthology.orchestra.run.smithi005.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-13T11:37:31.794 INFO:teuthology.orchestra.run.smithi005.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-13T11:37:31.794 INFO:teuthology.orchestra.run.smithi005.stdout:Recommended packages: 2024-09-13T11:37:31.794 INFO:teuthology.orchestra.run.smithi005.stdout: btrfs-tools 2024-09-13T11:37:31.819 INFO:teuthology.orchestra.run.smithi039.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-13T11:37:31.819 INFO:teuthology.orchestra.run.smithi039.stdout:Need to get 251 MB of archives. 2024-09-13T11:37:31.819 INFO:teuthology.orchestra.run.smithi039.stdout:After this operation, 933 MB of additional disk space will be used. 2024-09-13T11:37:31.819 INFO:teuthology.orchestra.run.smithi039.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-13T11:37:31.901 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4697-gd58f7306-1jammy [3,631 kB] 2024-09-13T11:37:31.903 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2024-09-13T11:37:31.903 INFO:teuthology.orchestra.run.smithi005.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T11:37:31.904 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T11:37:31.904 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-13T11:37:31.904 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-13T11:37:31.904 INFO:teuthology.orchestra.run.smithi005.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-13T11:37:31.904 INFO:teuthology.orchestra.run.smithi005.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-13T11:37:31.904 INFO:teuthology.orchestra.run.smithi005.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-13T11:37:31.905 INFO:teuthology.orchestra.run.smithi005.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-13T11:37:31.905 INFO:teuthology.orchestra.run.smithi005.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-13T11:37:31.905 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-13T11:37:31.905 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-13T11:37:31.905 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T11:37:31.905 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T11:37:31.905 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T11:37:31.905 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T11:37:31.905 INFO:teuthology.orchestra.run.smithi005.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T11:37:31.905 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T11:37:31.905 INFO:teuthology.orchestra.run.smithi005.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T11:37:31.905 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-13T11:37:31.906 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-13T11:37:31.906 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-13T11:37:31.906 INFO:teuthology.orchestra.run.smithi005.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-13T11:37:31.906 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-13T11:37:31.906 INFO:teuthology.orchestra.run.smithi005.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-13T11:37:31.906 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-13T11:37:31.906 INFO:teuthology.orchestra.run.smithi005.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-13T11:37:31.907 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-13T11:37:31.910 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-13T11:37:31.927 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-13T11:37:31.965 INFO:teuthology.orchestra.run.smithi005.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-13T11:37:31.965 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 251 MB of archives. 2024-09-13T11:37:31.965 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 933 MB of additional disk space will be used. 2024-09-13T11:37:31.965 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-13T11:37:31.986 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-13T11:37:31.991 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-13T11:37:32.003 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-13T11:37:32.008 INFO:teuthology.orchestra.run.smithi039.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-13T11:37:32.008 INFO:teuthology.orchestra.run.smithi039.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-13T11:37:32.009 INFO:teuthology.orchestra.run.smithi039.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-13T11:37:32.018 INFO:teuthology.orchestra.run.smithi039.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-13T11:37:32.020 INFO:teuthology.orchestra.run.smithi039.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-13T11:37:32.023 INFO:teuthology.orchestra.run.smithi039.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-13T11:37:32.024 INFO:teuthology.orchestra.run.smithi039.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-13T11:37:32.025 INFO:teuthology.orchestra.run.smithi039.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-13T11:37:32.027 INFO:teuthology.orchestra.run.smithi039.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-13T11:37:32.041 INFO:teuthology.orchestra.run.smithi039.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-13T11:37:32.043 INFO:teuthology.orchestra.run.smithi039.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-13T11:37:32.044 INFO:teuthology.orchestra.run.smithi039.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-13T11:37:32.044 INFO:teuthology.orchestra.run.smithi039.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-13T11:37:32.046 INFO:teuthology.orchestra.run.smithi039.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-13T11:37:32.049 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4697-gd58f7306-1jammy [3,631 kB] 2024-09-13T11:37:32.053 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-13T11:37:32.056 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-13T11:37:32.058 INFO:teuthology.orchestra.run.smithi039.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-13T11:37:32.059 INFO:teuthology.orchestra.run.smithi039.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-13T11:37:32.059 INFO:teuthology.orchestra.run.smithi039.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-13T11:37:32.060 INFO:teuthology.orchestra.run.smithi039.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-13T11:37:32.060 INFO:teuthology.orchestra.run.smithi039.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-13T11:37:32.073 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-13T11:37:32.075 INFO:teuthology.orchestra.run.smithi039.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-13T11:37:32.078 INFO:teuthology.orchestra.run.smithi039.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-13T11:37:32.079 INFO:teuthology.orchestra.run.smithi039.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-13T11:37:32.079 INFO:teuthology.orchestra.run.smithi039.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-13T11:37:32.080 INFO:teuthology.orchestra.run.smithi039.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-13T11:37:32.093 INFO:teuthology.orchestra.run.smithi039.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T11:37:32.093 INFO:teuthology.orchestra.run.smithi039.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-13T11:37:32.097 INFO:teuthology.orchestra.run.smithi039.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-13T11:37:32.097 INFO:teuthology.orchestra.run.smithi039.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-13T11:37:32.097 INFO:teuthology.orchestra.run.smithi039.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-13T11:37:32.109 INFO:teuthology.orchestra.run.smithi039.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4697-gd58f7306-1jammy [3,320 kB] 2024-09-13T11:37:32.109 INFO:teuthology.orchestra.run.smithi039.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-13T11:37:32.115 INFO:teuthology.orchestra.run.smithi039.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-13T11:37:32.115 INFO:teuthology.orchestra.run.smithi039.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-13T11:37:32.116 INFO:teuthology.orchestra.run.smithi039.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-13T11:37:32.117 INFO:teuthology.orchestra.run.smithi039.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-13T11:37:32.127 INFO:teuthology.orchestra.run.smithi039.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-13T11:37:32.129 INFO:teuthology.orchestra.run.smithi039.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-13T11:37:32.130 INFO:teuthology.orchestra.run.smithi039.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-13T11:37:32.131 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-13T11:37:32.132 INFO:teuthology.orchestra.run.smithi039.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-13T11:37:32.132 INFO:teuthology.orchestra.run.smithi039.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-13T11:37:32.136 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-13T11:37:32.144 INFO:teuthology.orchestra.run.smithi039.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-13T11:37:32.145 INFO:teuthology.orchestra.run.smithi039.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-13T11:37:32.145 INFO:teuthology.orchestra.run.smithi039.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-13T11:37:32.148 INFO:teuthology.orchestra.run.smithi039.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-13T11:37:32.149 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-13T11:37:32.153 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-13T11:37:32.153 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-13T11:37:32.154 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-13T11:37:32.163 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-13T11:37:32.166 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-13T11:37:32.168 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-13T11:37:32.169 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-13T11:37:32.171 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-13T11:37:32.173 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-13T11:37:32.187 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-13T11:37:32.188 INFO:teuthology.orchestra.run.smithi039.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4697-gd58f7306-1jammy [722 kB] 2024-09-13T11:37:32.189 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-13T11:37:32.190 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-13T11:37:32.190 INFO:teuthology.orchestra.run.smithi005.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-13T11:37:32.191 INFO:teuthology.orchestra.run.smithi005.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-13T11:37:32.193 INFO:teuthology.orchestra.run.smithi005.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-13T11:37:32.193 INFO:teuthology.orchestra.run.smithi005.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-13T11:37:32.193 INFO:teuthology.orchestra.run.smithi005.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-13T11:37:32.201 INFO:teuthology.orchestra.run.smithi039.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-13T11:37:32.202 INFO:teuthology.orchestra.run.smithi039.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-13T11:37:32.203 INFO:teuthology.orchestra.run.smithi039.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T11:37:32.203 INFO:teuthology.orchestra.run.smithi039.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-13T11:37:32.205 INFO:teuthology.orchestra.run.smithi005.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-13T11:37:32.206 INFO:teuthology.orchestra.run.smithi005.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-13T11:37:32.214 INFO:teuthology.orchestra.run.smithi039.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4697-gd58f7306-1jammy [356 kB] 2024-09-13T11:37:32.214 INFO:teuthology.orchestra.run.smithi039.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-13T11:37:32.220 INFO:teuthology.orchestra.run.smithi039.stdout:Get:60 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4697-gd58f7306-1jammy [32.8 kB] 2024-09-13T11:37:32.221 INFO:teuthology.orchestra.run.smithi039.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4697-gd58f7306-1jammy [180 kB] 2024-09-13T11:37:32.222 INFO:teuthology.orchestra.run.smithi005.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-13T11:37:32.225 INFO:teuthology.orchestra.run.smithi039.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4697-gd58f7306-1jammy [71.2 kB] 2024-09-13T11:37:32.225 INFO:teuthology.orchestra.run.smithi005.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-13T11:37:32.226 INFO:teuthology.orchestra.run.smithi005.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-13T11:37:32.226 INFO:teuthology.orchestra.run.smithi005.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-13T11:37:32.226 INFO:teuthology.orchestra.run.smithi005.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-13T11:37:32.227 INFO:teuthology.orchestra.run.smithi039.stdout:Get:63 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4697-gd58f7306-1jammy [338 kB] 2024-09-13T11:37:32.227 INFO:teuthology.orchestra.run.smithi005.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T11:37:32.228 INFO:teuthology.orchestra.run.smithi005.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-13T11:37:32.228 INFO:teuthology.orchestra.run.smithi005.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-13T11:37:32.232 INFO:teuthology.orchestra.run.smithi039.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-13T11:37:32.233 INFO:teuthology.orchestra.run.smithi039.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4697-gd58f7306-1jammy [7,979 kB] 2024-09-13T11:37:32.234 INFO:teuthology.orchestra.run.smithi039.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-13T11:37:32.234 INFO:teuthology.orchestra.run.smithi039.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-13T11:37:32.239 INFO:teuthology.orchestra.run.smithi005.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-13T11:37:32.240 INFO:teuthology.orchestra.run.smithi005.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-13T11:37:32.251 INFO:teuthology.orchestra.run.smithi039.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-13T11:37:32.251 INFO:teuthology.orchestra.run.smithi039.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-13T11:37:32.251 INFO:teuthology.orchestra.run.smithi039.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-13T11:37:32.252 INFO:teuthology.orchestra.run.smithi039.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-13T11:37:32.252 INFO:teuthology.orchestra.run.smithi039.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-13T11:37:32.253 INFO:teuthology.orchestra.run.smithi039.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-13T11:37:32.255 INFO:teuthology.orchestra.run.smithi039.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-13T11:37:32.255 INFO:teuthology.orchestra.run.smithi039.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-13T11:37:32.257 INFO:teuthology.orchestra.run.smithi005.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-13T11:37:32.259 INFO:teuthology.orchestra.run.smithi039.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-13T11:37:32.262 INFO:teuthology.orchestra.run.smithi005.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-13T11:37:32.263 INFO:teuthology.orchestra.run.smithi005.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-13T11:37:32.263 INFO:teuthology.orchestra.run.smithi005.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-13T11:37:32.264 INFO:teuthology.orchestra.run.smithi005.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-13T11:37:32.265 INFO:teuthology.orchestra.run.smithi005.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-13T11:37:32.267 INFO:teuthology.orchestra.run.smithi005.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-13T11:37:32.268 INFO:teuthology.orchestra.run.smithi005.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-13T11:37:32.272 INFO:teuthology.orchestra.run.smithi039.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-13T11:37:32.273 INFO:teuthology.orchestra.run.smithi039.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-13T11:37:32.274 INFO:teuthology.orchestra.run.smithi039.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-13T11:37:32.274 INFO:teuthology.orchestra.run.smithi005.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-13T11:37:32.275 INFO:teuthology.orchestra.run.smithi005.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-13T11:37:32.277 INFO:teuthology.orchestra.run.smithi039.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-13T11:37:32.280 INFO:teuthology.orchestra.run.smithi039.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-13T11:37:32.280 INFO:teuthology.orchestra.run.smithi039.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-13T11:37:32.281 INFO:teuthology.orchestra.run.smithi039.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-13T11:37:32.286 INFO:teuthology.orchestra.run.smithi005.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4697-gd58f7306-1jammy [3,320 kB] 2024-09-13T11:37:32.289 INFO:teuthology.orchestra.run.smithi039.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-13T11:37:32.290 INFO:teuthology.orchestra.run.smithi039.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-13T11:37:32.291 INFO:teuthology.orchestra.run.smithi005.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-13T11:37:32.292 INFO:teuthology.orchestra.run.smithi005.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-13T11:37:32.293 INFO:teuthology.orchestra.run.smithi039.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-13T11:37:32.293 INFO:teuthology.orchestra.run.smithi005.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-13T11:37:32.295 INFO:teuthology.orchestra.run.smithi005.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-13T11:37:32.306 INFO:teuthology.orchestra.run.smithi039.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T11:37:32.307 INFO:teuthology.orchestra.run.smithi039.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-13T11:37:32.307 INFO:teuthology.orchestra.run.smithi039.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-13T11:37:32.309 INFO:teuthology.orchestra.run.smithi039.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-13T11:37:32.309 INFO:teuthology.orchestra.run.smithi039.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-13T11:37:32.317 INFO:teuthology.orchestra.run.smithi039.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-13T11:37:32.317 INFO:teuthology.orchestra.run.smithi039.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-13T11:37:32.323 INFO:teuthology.orchestra.run.smithi039.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-13T11:37:32.325 INFO:teuthology.orchestra.run.smithi039.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-13T11:37:32.326 INFO:teuthology.orchestra.run.smithi039.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-13T11:37:32.350 INFO:teuthology.orchestra.run.smithi005.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-13T11:37:32.351 INFO:teuthology.orchestra.run.smithi005.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-13T11:37:32.353 INFO:teuthology.orchestra.run.smithi005.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T11:37:32.353 INFO:teuthology.orchestra.run.smithi005.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-13T11:37:32.354 INFO:teuthology.orchestra.run.smithi005.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-13T11:37:32.372 INFO:teuthology.orchestra.run.smithi005.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-13T11:37:32.374 INFO:teuthology.orchestra.run.smithi005.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-13T11:37:32.374 INFO:teuthology.orchestra.run.smithi005.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-13T11:37:32.388 INFO:teuthology.orchestra.run.smithi005.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-13T11:37:32.388 INFO:teuthology.orchestra.run.smithi005.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-13T11:37:32.389 INFO:teuthology.orchestra.run.smithi005.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-13T11:37:32.389 INFO:teuthology.orchestra.run.smithi005.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-13T11:37:32.390 INFO:teuthology.orchestra.run.smithi005.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-13T11:37:32.390 INFO:teuthology.orchestra.run.smithi005.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-13T11:37:32.401 INFO:teuthology.orchestra.run.smithi005.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4697-gd58f7306-1jammy [722 kB] 2024-09-13T11:37:32.404 INFO:teuthology.orchestra.run.smithi005.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-13T11:37:32.405 INFO:teuthology.orchestra.run.smithi005.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-13T11:37:32.408 INFO:teuthology.orchestra.run.smithi005.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-13T11:37:32.421 INFO:teuthology.orchestra.run.smithi005.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-13T11:37:32.422 INFO:teuthology.orchestra.run.smithi005.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-13T11:37:32.423 INFO:teuthology.orchestra.run.smithi005.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-13T11:37:32.426 INFO:teuthology.orchestra.run.smithi005.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-13T11:37:32.428 INFO:teuthology.orchestra.run.smithi005.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-13T11:37:32.428 INFO:teuthology.orchestra.run.smithi005.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-13T11:37:32.429 INFO:teuthology.orchestra.run.smithi005.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-13T11:37:32.434 INFO:teuthology.orchestra.run.smithi005.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4697-gd58f7306-1jammy [356 kB] 2024-09-13T11:37:32.438 INFO:teuthology.orchestra.run.smithi005.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-13T11:37:32.439 INFO:teuthology.orchestra.run.smithi005.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-13T11:37:32.442 INFO:teuthology.orchestra.run.smithi005.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-13T11:37:32.449 INFO:teuthology.orchestra.run.smithi039.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4697-gd58f7306-1jammy [112 kB] 2024-09-13T11:37:32.452 INFO:teuthology.orchestra.run.smithi039.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4697-gd58f7306-1jammy [473 kB] 2024-09-13T11:37:32.456 INFO:teuthology.orchestra.run.smithi005.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T11:37:32.457 INFO:teuthology.orchestra.run.smithi005.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-13T11:37:32.457 INFO:teuthology.orchestra.run.smithi005.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-13T11:37:32.458 INFO:teuthology.orchestra.run.smithi005.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-13T11:37:32.459 INFO:teuthology.orchestra.run.smithi005.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-13T11:37:32.461 INFO:teuthology.orchestra.run.smithi005.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4697-gd58f7306-1jammy [32.8 kB] 2024-09-13T11:37:32.462 INFO:teuthology.orchestra.run.smithi039.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4697-gd58f7306-1jammy [28.2 MB] 2024-09-13T11:37:32.463 INFO:teuthology.orchestra.run.smithi005.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4697-gd58f7306-1jammy [180 kB] 2024-09-13T11:37:32.465 INFO:teuthology.orchestra.run.smithi005.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-13T11:37:32.465 INFO:teuthology.orchestra.run.smithi005.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-13T11:37:32.466 INFO:teuthology.orchestra.run.smithi005.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4697-gd58f7306-1jammy [71.2 kB] 2024-09-13T11:37:32.468 INFO:teuthology.orchestra.run.smithi005.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4697-gd58f7306-1jammy [338 kB] 2024-09-13T11:37:32.473 INFO:teuthology.orchestra.run.smithi005.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-13T11:37:32.475 INFO:teuthology.orchestra.run.smithi005.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-13T11:37:32.475 INFO:teuthology.orchestra.run.smithi005.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-13T11:37:32.476 INFO:teuthology.orchestra.run.smithi005.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4697-gd58f7306-1jammy [7,979 kB] 2024-09-13T11:37:32.802 INFO:teuthology.orchestra.run.smithi005.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4697-gd58f7306-1jammy [112 kB] 2024-09-13T11:37:32.804 INFO:teuthology.orchestra.run.smithi005.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4697-gd58f7306-1jammy [473 kB] 2024-09-13T11:37:32.814 INFO:teuthology.orchestra.run.smithi005.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4697-gd58f7306-1jammy [28.2 MB] 2024-09-13T11:37:33.347 INFO:teuthology.orchestra.run.smithi039.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4697-gd58f7306-1jammy [5,206 kB] 2024-09-13T11:37:33.450 INFO:teuthology.orchestra.run.smithi039.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4697-gd58f7306-1jammy [251 kB] 2024-09-13T11:37:33.453 INFO:teuthology.orchestra.run.smithi039.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4697-gd58f7306-1jammy [124 kB] 2024-09-13T11:37:33.455 INFO:teuthology.orchestra.run.smithi039.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4697-gd58f7306-1jammy [1,561 kB] 2024-09-13T11:37:33.587 INFO:teuthology.orchestra.run.smithi039.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4697-gd58f7306-1jammy [6,309 kB] 2024-09-13T11:37:33.702 INFO:teuthology.orchestra.run.smithi039.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4697-gd58f7306-1jammy [23.1 MB] 2024-09-13T11:37:33.936 INFO:teuthology.orchestra.run.smithi005.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4697-gd58f7306-1jammy [5,206 kB] 2024-09-13T11:37:34.077 INFO:teuthology.orchestra.run.smithi005.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4697-gd58f7306-1jammy [251 kB] 2024-09-13T11:37:34.082 INFO:teuthology.orchestra.run.smithi005.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4697-gd58f7306-1jammy [124 kB] 2024-09-13T11:37:34.085 INFO:teuthology.orchestra.run.smithi005.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4697-gd58f7306-1jammy [1,561 kB] 2024-09-13T11:37:34.164 INFO:teuthology.orchestra.run.smithi005.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4697-gd58f7306-1jammy [6,309 kB] 2024-09-13T11:37:34.389 INFO:teuthology.orchestra.run.smithi039.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4697-gd58f7306-1jammy [14.1 kB] 2024-09-13T11:37:34.389 INFO:teuthology.orchestra.run.smithi039.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4697-gd58f7306-1jammy [880 kB] 2024-09-13T11:37:34.401 INFO:teuthology.orchestra.run.smithi005.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4697-gd58f7306-1jammy [23.1 MB] 2024-09-13T11:37:34.402 INFO:teuthology.orchestra.run.smithi039.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4697-gd58f7306-1jammy [2,433 kB] 2024-09-13T11:37:34.439 INFO:teuthology.orchestra.run.smithi039.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4697-gd58f7306-1jammy [812 kB] 2024-09-13T11:37:34.455 INFO:teuthology.orchestra.run.smithi039.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4697-gd58f7306-1jammy [160 kB] 2024-09-13T11:37:34.458 INFO:teuthology.orchestra.run.smithi039.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4697-gd58f7306-1jammy [61.4 MB] 2024-09-13T11:37:35.422 INFO:teuthology.orchestra.run.smithi005.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4697-gd58f7306-1jammy [14.1 kB] 2024-09-13T11:37:35.423 INFO:teuthology.orchestra.run.smithi005.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4697-gd58f7306-1jammy [880 kB] 2024-09-13T11:37:35.440 INFO:teuthology.orchestra.run.smithi005.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4697-gd58f7306-1jammy [2,433 kB] 2024-09-13T11:37:35.484 INFO:teuthology.orchestra.run.smithi005.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4697-gd58f7306-1jammy [812 kB] 2024-09-13T11:37:35.510 INFO:teuthology.orchestra.run.smithi005.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4697-gd58f7306-1jammy [160 kB] 2024-09-13T11:37:35.513 INFO:teuthology.orchestra.run.smithi005.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4697-gd58f7306-1jammy [61.4 MB] 2024-09-13T11:37:36.469 INFO:teuthology.orchestra.run.smithi039.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4697-gd58f7306-1jammy [8,625 kB] 2024-09-13T11:37:36.653 INFO:teuthology.orchestra.run.smithi039.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4697-gd58f7306-1jammy [14.2 kB] 2024-09-13T11:37:36.654 INFO:teuthology.orchestra.run.smithi039.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4697-gd58f7306-1jammy [55.7 MB] 2024-09-13T11:37:37.813 INFO:teuthology.orchestra.run.smithi005.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4697-gd58f7306-1jammy [8,625 kB] 2024-09-13T11:37:38.041 INFO:teuthology.orchestra.run.smithi005.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4697-gd58f7306-1jammy [14.2 kB] 2024-09-13T11:37:38.041 INFO:teuthology.orchestra.run.smithi005.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4697-gd58f7306-1jammy [55.7 MB] 2024-09-13T11:37:38.380 INFO:teuthology.orchestra.run.smithi039.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4697-gd58f7306-1jammy [130 kB] 2024-09-13T11:37:38.383 INFO:teuthology.orchestra.run.smithi039.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4697-gd58f7306-1jammy [38.9 kB] 2024-09-13T11:37:38.384 INFO:teuthology.orchestra.run.smithi039.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4697-gd58f7306-1jammy [15.9 MB] 2024-09-13T11:37:38.860 INFO:teuthology.orchestra.run.smithi039.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4697-gd58f7306-1jammy [91.3 kB] 2024-09-13T11:37:39.460 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 251 MB in 7s (35.4 MB/s) 2024-09-13T11:37:39.520 INFO:teuthology.orchestra.run.smithi005.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4697-gd58f7306-1jammy [130 kB] 2024-09-13T11:37:39.522 INFO:teuthology.orchestra.run.smithi005.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4697-gd58f7306-1jammy [38.9 kB] 2024-09-13T11:37:39.563 INFO:teuthology.orchestra.run.smithi005.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4697-gd58f7306-1jammy [15.9 MB] 2024-09-13T11:37:39.596 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-13T11:37:39.697 INFO:teuthology.orchestra.run.smithi039.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-13T11:37:39.702 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-13T11:37:39.769 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T11:37:39.872 INFO:teuthology.orchestra.run.smithi005.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4697-gd58f7306-1jammy [91.3 kB] 2024-09-13T11:37:40.089 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T11:37:40.116 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-13T11:37:40.156 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T11:37:40.476 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 251 MB in 8s (31.5 MB/s) 2024-09-13T11:37:40.525 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T11:37:40.552 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-13T11:37:40.592 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T11:37:40.603 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-13T11:37:40.705 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-13T11:37:40.711 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-13T11:37:40.785 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T11:37:41.037 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T11:37:41.064 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:37:41.072 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T11:37:41.099 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-13T11:37:41.111 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:37:41.139 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T11:37:41.457 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T11:37:41.475 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T11:37:41.483 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:37:41.502 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-13T11:37:41.523 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:37:41.542 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T11:37:41.885 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T11:37:41.911 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:37:41.951 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:37:41.978 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T11:37:42.006 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:37:42.053 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:37:42.313 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-13T11:37:42.339 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-13T11:37:42.388 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T11:37:42.423 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T11:37:42.450 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:37:42.490 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:37:42.724 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librados2. 2024-09-13T11:37:42.751 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../007-librados2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:42.790 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:42.826 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T11:37:42.853 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T11:37:42.893 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:37:43.229 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-13T11:37:43.256 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-13T11:37:43.296 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T11:37:43.310 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librbd1. 2024-09-13T11:37:43.337 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../008-librbd1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:43.377 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:43.565 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librados2. 2024-09-13T11:37:43.578 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../007-librados2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:43.612 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:43.905 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T11:37:43.931 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../009-libcephfs2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:43.979 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:44.193 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librbd1. 2024-09-13T11:37:44.220 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../008-librbd1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:44.259 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:44.290 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rados. 2024-09-13T11:37:44.317 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../010-python3-rados_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:44.357 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:44.659 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T11:37:44.686 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:37:44.695 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T11:37:44.722 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../009-libcephfs2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:44.726 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:44.762 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:44.998 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rados. 2024-09-13T11:37:45.014 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../010-python3-rados_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:45.048 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:45.342 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T11:37:45.345 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T11:37:45.369 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:37:45.372 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../012-python3-cephfs_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:45.409 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:45.420 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:45.670 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T11:37:45.685 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../012-python3-cephfs_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:45.719 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:45.773 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T11:37:45.799 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../013-python3-ceph-common_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:37:45.848 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:45.989 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T11:37:46.005 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../013-python3-ceph-common_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:37:46.047 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:46.234 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T11:37:46.261 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-13T11:37:46.301 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T11:37:46.350 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T11:37:46.377 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-13T11:37:46.417 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T11:37:46.620 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T11:37:46.647 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-13T11:37:46.695 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-13T11:37:46.728 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T11:37:46.755 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-13T11:37:46.794 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-13T11:37:46.965 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T11:37:46.991 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../016-python3-rbd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:47.031 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:47.047 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T11:37:47.074 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../016-python3-rbd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:47.114 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:47.408 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T11:37:47.435 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-13T11:37:47.459 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T11:37:47.466 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T11:37:47.486 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-13T11:37:47.534 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T11:37:47.761 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-13T11:37:47.788 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-13T11:37:47.827 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T11:37:47.861 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-13T11:37:47.888 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-13T11:37:47.928 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T11:37:48.146 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-13T11:37:48.163 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-13T11:37:48.205 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T11:37:48.247 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-13T11:37:48.263 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-13T11:37:48.306 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T11:37:48.500 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package lua5.1. 2024-09-13T11:37:48.527 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-13T11:37:48.566 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-13T11:37:48.676 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua5.1. 2024-09-13T11:37:48.704 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-13T11:37:48.751 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-13T11:37:49.003 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package lua-any. 2024-09-13T11:37:49.030 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-13T11:37:49.069 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-13T11:37:49.120 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua-any. 2024-09-13T11:37:49.147 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-13T11:37:49.220 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-13T11:37:49.313 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package zip. 2024-09-13T11:37:49.330 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-13T11:37:49.372 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking zip (3.0-12build2) ... 2024-09-13T11:37:49.473 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package zip. 2024-09-13T11:37:49.489 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-13T11:37:49.540 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking zip (3.0-12build2) ... 2024-09-13T11:37:49.700 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package unzip. 2024-09-13T11:37:49.727 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-13T11:37:49.767 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-13T11:37:49.851 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package unzip. 2024-09-13T11:37:49.878 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-13T11:37:49.918 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-13T11:37:50.096 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package luarocks. 2024-09-13T11:37:50.112 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-13T11:37:50.145 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-13T11:37:50.520 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librgw2. 2024-09-13T11:37:50.522 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package luarocks. 2024-09-13T11:37:50.547 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../025-librgw2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:50.549 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-13T11:37:50.586 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:50.588 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-13T11:37:50.946 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librgw2. 2024-09-13T11:37:50.973 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../025-librgw2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:51.013 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:51.213 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T11:37:51.230 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../026-python3-rgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:51.272 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:51.583 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-13T11:37:51.611 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-13T11:37:51.632 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T11:37:51.649 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T11:37:51.659 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../026-python3-rgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:51.698 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:51.928 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T11:37:51.955 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../028-libradosstriper1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:51.994 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:52.043 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-13T11:37:52.070 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-13T11:37:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T11:37:52.371 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T11:37:52.397 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../028-libradosstriper1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:52.422 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-common. 2024-09-13T11:37:52.429 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:52.449 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../029-ceph-common_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:52.489 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:52.866 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-common. 2024-09-13T11:37:52.893 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../029-ceph-common_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:52.932 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:53.927 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-base. 2024-09-13T11:37:53.954 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../030-ceph-base_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:53.995 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:54.562 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-base. 2024-09-13T11:37:54.590 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../030-ceph-base_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:37:54.647 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:37:54.673 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-13T11:37:54.700 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-13T11:37:54.740 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-13T11:37:55.026 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cheroot. 2024-09-13T11:37:55.053 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-13T11:37:55.092 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T11:37:55.175 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-13T11:37:55.202 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-13T11:37:55.242 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-13T11:37:55.404 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-13T11:37:55.431 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-13T11:37:55.470 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-13T11:37:55.544 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cheroot. 2024-09-13T11:37:55.571 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-13T11:37:55.611 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T11:37:55.748 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-13T11:37:55.775 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-13T11:37:55.806 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-13T11:37:55.939 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-13T11:37:55.966 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-13T11:37:56.006 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-13T11:37:56.109 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-13T11:37:56.136 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-13T11:37:56.176 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-13T11:37:56.267 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-13T11:37:56.283 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-13T11:37:56.316 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-13T11:37:56.478 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-tempora. 2024-09-13T11:37:56.506 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-13T11:37:56.545 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-13T11:37:56.578 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-13T11:37:56.605 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-13T11:37:56.644 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-13T11:37:56.848 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-portend. 2024-09-13T11:37:56.875 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-13T11:37:56.914 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-13T11:37:56.949 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-tempora. 2024-09-13T11:37:56.977 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-13T11:37:57.016 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-13T11:37:57.176 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-13T11:37:57.203 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-13T11:37:57.234 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-13T11:37:57.277 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-portend. 2024-09-13T11:37:57.304 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-13T11:37:57.335 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-13T11:37:57.520 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T11:37:57.536 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-13T11:37:57.578 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-13T11:37:57.621 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-13T11:37:57.649 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-13T11:37:57.688 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-13T11:37:57.892 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T11:37:57.906 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-13T11:37:57.940 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-13T11:37:58.016 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T11:37:58.044 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-13T11:37:58.083 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-13T11:37:58.245 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T11:37:58.272 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-13T11:37:58.311 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-13T11:37:58.420 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T11:37:58.447 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-13T11:37:58.486 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-13T11:37:58.598 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T11:37:58.625 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-13T11:37:58.664 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-13T11:37:58.799 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T11:37:58.825 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-13T11:37:58.864 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-13T11:37:58.908 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-mako. 2024-09-13T11:37:58.925 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-13T11:37:58.958 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T11:37:59.168 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T11:37:59.194 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-13T11:37:59.234 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-13T11:37:59.270 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T11:37:59.297 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T11:37:59.336 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T11:37:59.487 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-mako. 2024-09-13T11:37:59.514 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-13T11:37:59.545 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T11:37:59.639 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T11:37:59.667 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-13T11:37:59.707 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-13T11:37:59.907 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T11:37:59.935 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T11:37:59.982 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T11:38:00.034 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webob. 2024-09-13T11:38:00.061 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-13T11:38:00.100 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T11:38:00.285 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T11:38:00.313 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-13T11:38:00.352 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-13T11:38:00.470 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T11:38:00.497 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-13T11:38:00.540 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T11:38:00.696 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-webob. 2024-09-13T11:38:00.724 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-13T11:38:00.856 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T11:38:00.883 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-13T11:38:00.888 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T11:38:00.922 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T11:38:01.192 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-paste. 2024-09-13T11:38:01.208 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-13T11:38:01.242 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T11:38:01.317 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T11:38:01.344 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-13T11:38:01.394 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T11:38:01.578 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T11:38:01.606 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-13T11:38:01.645 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T11:38:01.703 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T11:38:01.730 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-13T11:38:01.770 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T11:38:01.881 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T11:38:01.897 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-13T11:38:01.931 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-13T11:38:02.089 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-paste. 2024-09-13T11:38:02.116 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-13T11:38:02.172 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T11:38:02.234 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T11:38:02.261 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-13T11:38:02.300 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-13T11:38:02.501 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T11:38:02.517 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-13T11:38:02.559 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T11:38:02.620 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T11:38:02.647 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-13T11:38:02.686 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T11:38:02.862 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T11:38:02.889 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-13T11:38:02.928 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-13T11:38:03.058 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-certifi. 2024-09-13T11:38:03.085 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-13T11:38:03.124 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-13T11:38:03.273 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T11:38:03.300 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-13T11:38:03.340 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-13T11:38:03.452 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-idna. 2024-09-13T11:38:03.479 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-13T11:38:03.527 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T11:38:03.618 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T11:38:03.645 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-13T11:38:03.676 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T11:38:03.780 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-urllib3. 2024-09-13T11:38:03.808 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-13T11:38:03.838 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T11:38:04.006 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-certifi. 2024-09-13T11:38:04.034 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-13T11:38:04.064 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-13T11:38:04.141 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-requests. 2024-09-13T11:38:04.168 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-13T11:38:04.199 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T11:38:04.384 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-idna. 2024-09-13T11:38:04.411 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-13T11:38:04.451 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T11:38:04.494 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T11:38:04.521 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-13T11:38:04.569 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T11:38:04.771 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-urllib3. 2024-09-13T11:38:04.798 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-13T11:38:04.830 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T11:38:04.846 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:38:05.105 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:05.212 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T11:38:05.933 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-requests. 2024-09-13T11:38:05.960 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-13T11:38:05.999 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T11:38:06.018 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T11:38:06.039 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:06.076 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:06.285 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T11:38:06.302 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-13T11:38:06.335 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T11:38:06.379 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T11:38:06.407 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-13T11:38:06.446 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-13T11:38:06.639 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T11:38:06.666 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:38:06.705 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:06.707 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T11:38:06.723 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-13T11:38:06.757 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-13T11:38:07.026 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T11:38:07.043 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../063-ceph-mgr_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:07.076 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:07.102 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T11:38:07.130 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:07.169 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:07.446 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T11:38:07.473 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../064-ceph-mon_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:07.488 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T11:38:07.513 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:07.515 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-13T11:38:07.555 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-13T11:38:07.824 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T11:38:07.852 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-13T11:38:07.882 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-13T11:38:08.133 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T11:38:08.160 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../065-ceph-osd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:08.219 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T11:38:08.224 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:08.247 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../063-ceph-mgr_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:08.286 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:08.689 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T11:38:08.717 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../064-ceph-mon_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:08.755 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:09.294 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph. 2024-09-13T11:38:09.310 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../066-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:09.318 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T11:38:09.345 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../065-ceph-osd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:09.352 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:09.376 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:09.763 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T11:38:09.791 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../067-ceph-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:09.830 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:10.183 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T11:38:10.211 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../068-ceph-mds_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:10.249 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:10.463 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph. 2024-09-13T11:38:10.479 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../066-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:10.521 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:10.660 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package cephadm. 2024-09-13T11:38:10.688 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../069-cephadm_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:10.765 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:10.857 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T11:38:10.885 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../067-ceph-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:10.923 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:11.038 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T11:38:11.066 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-13T11:38:11.096 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-13T11:38:11.399 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-13T11:38:11.402 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T11:38:11.426 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T11:38:11.430 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../068-ceph-mds_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:11.460 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:11.465 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T11:38:11.838 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package cephadm. 2024-09-13T11:38:11.866 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../069-cephadm_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:11.905 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:12.208 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T11:38:12.235 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-13T11:38:12.274 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-13T11:38:12.487 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-babel. 2024-09-13T11:38:12.515 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T11:38:12.554 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T11:38:12.636 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-13T11:38:12.663 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T11:38:12.702 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T11:38:12.881 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T11:38:12.898 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-13T11:38:12.939 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T11:38:13.293 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T11:38:13.320 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:38:13.359 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:13.696 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T11:38:13.723 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T11:38:13.762 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T11:38:13.783 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-babel. 2024-09-13T11:38:13.810 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T11:38:13.849 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T11:38:14.040 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-routes. 2024-09-13T11:38:14.067 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-13T11:38:14.106 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T11:38:14.186 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T11:38:14.213 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-13T11:38:14.252 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T11:38:14.435 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T11:38:14.462 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:38:14.501 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:14.597 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T11:38:14.624 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:38:14.664 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:14.985 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T11:38:15.012 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T11:38:15.051 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T11:38:15.370 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-routes. 2024-09-13T11:38:15.397 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-13T11:38:15.437 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T11:38:15.782 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T11:38:15.809 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:38:15.848 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:17.514 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-13T11:38:17.533 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-13T11:38:17.572 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T11:38:18.059 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T11:38:18.087 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-13T11:38:18.126 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T11:38:18.479 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-13T11:38:18.507 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-13T11:38:18.545 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-13T11:38:18.746 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-13T11:38:18.777 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-13T11:38:18.820 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T11:38:18.848 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T11:38:18.876 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-13T11:38:18.915 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T11:38:19.316 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T11:38:19.332 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-13T11:38:19.365 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T11:38:19.615 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T11:38:19.643 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:38:19.753 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-13T11:38:19.758 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:19.781 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-13T11:38:19.819 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-13T11:38:20.130 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T11:38:20.158 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-13T11:38:20.197 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T11:38:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T11:38:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-13T11:38:20.729 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-13T11:38:20.948 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T11:38:20.976 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:38:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T11:38:21.026 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-13T11:38:21.065 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-13T11:38:21.108 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T11:38:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-13T11:38:21.443 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T11:38:21.738 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T11:38:21.766 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-13T11:38:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-13T11:38:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T11:38:22.153 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-13T11:38:22.191 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-13T11:38:22.262 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T11:38:22.290 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-13T11:38:22.329 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-13T11:38:22.511 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T11:38:22.539 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-13T11:38:22.577 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T11:38:22.632 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T11:38:22.660 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-13T11:38:22.698 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-13T11:38:22.864 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T11:38:22.880 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-13T11:38:22.922 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-13T11:38:23.019 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T11:38:23.047 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-13T11:38:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T11:38:23.183 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T11:38:23.200 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-13T11:38:23.236 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T11:38:23.405 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T11:38:23.433 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-13T11:38:23.463 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-13T11:38:23.775 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T11:38:23.804 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-13T11:38:23.851 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-13T11:38:23.897 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T11:38:23.914 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:38:23.955 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:24.187 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T11:38:24.216 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-13T11:38:24.250 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T11:38:24.253 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T11:38:24.278 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-13T11:38:24.308 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T11:38:24.565 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T11:38:24.593 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-13T11:38:24.631 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-13T11:38:24.661 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T11:38:24.690 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T11:38:24.727 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T11:38:24.977 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T11:38:24.997 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package jq. 2024-09-13T11:38:25.005 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-13T11:38:25.025 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T11:38:25.043 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T11:38:25.063 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-13T11:38:25.316 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package socat. 2024-09-13T11:38:25.345 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-13T11:38:25.382 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-13T11:38:25.677 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T11:38:25.694 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-13T11:38:25.716 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T11:38:25.735 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-13T11:38:25.744 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:38:25.782 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:26.089 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-test. 2024-09-13T11:38:26.106 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../097-ceph-test_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:26.119 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T11:38:26.147 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-13T11:38:26.147 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:26.185 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T11:38:26.664 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T11:38:26.692 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T11:38:26.730 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T11:38:26.966 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package jq. 2024-09-13T11:38:26.982 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T11:38:27.024 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-13T11:38:27.277 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package socat. 2024-09-13T11:38:27.294 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-13T11:38:27.335 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-13T11:38:27.647 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T11:38:27.676 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-13T11:38:27.714 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-13T11:38:28.077 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-test. 2024-09-13T11:38:28.093 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../097-ceph-test_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:28.126 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:29.078 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T11:38:29.107 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../098-ceph-volume_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:38:29.144 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:29.449 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T11:38:29.477 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../099-libcephfs-dev_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:29.515 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:29.834 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-13T11:38:29.863 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-13T11:38:29.900 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T11:38:30.188 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-13T11:38:30.217 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-13T11:38:30.255 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-13T11:38:30.592 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T11:38:30.614 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T11:38:30.621 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-13T11:38:30.642 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../098-ceph-volume_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T11:38:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T11:38:30.680 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:30.976 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T11:38:30.993 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../099-libcephfs-dev_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:31.035 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:31.089 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package pkg-config. 2024-09-13T11:38:31.117 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-13T11:38:31.155 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T11:38:31.355 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-13T11:38:31.384 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-13T11:38:31.416 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T11:38:31.421 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T11:38:31.433 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-13T11:38:31.474 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-13T11:38:31.751 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-13T11:38:31.779 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-13T11:38:31.817 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-13T11:38:31.903 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-13T11:38:31.931 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-13T11:38:31.969 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-13T11:38:32.163 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T11:38:32.192 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-13T11:38:32.229 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T11:38:32.238 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T11:38:32.267 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T11:38:32.297 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T11:38:32.592 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T11:38:32.620 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-13T11:38:32.643 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package pkg-config. 2024-09-13T11:38:32.658 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-13T11:38:32.672 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-13T11:38:32.725 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T11:38:32.988 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T11:38:32.994 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T11:38:33.016 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-13T11:38:33.023 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-13T11:38:33.054 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-13T11:38:33.060 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-13T11:38:33.397 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-py. 2024-09-13T11:38:33.426 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-13T11:38:33.463 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-13T11:38:33.483 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-13T11:38:33.511 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-13T11:38:33.549 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-13T11:38:33.784 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pygments. 2024-09-13T11:38:33.812 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-13T11:38:33.850 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T11:38:33.861 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T11:38:33.889 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T11:38:33.927 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T11:38:34.214 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T11:38:34.231 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-13T11:38:34.264 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-13T11:38:34.321 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T11:38:34.350 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-13T11:38:34.387 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-13T11:38:34.567 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T11:38:34.595 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-13T11:38:34.633 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-13T11:38:34.681 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-toml. 2024-09-13T11:38:34.710 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-13T11:38:34.748 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-13T11:38:35.003 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-py. 2024-09-13T11:38:35.032 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-13T11:38:35.051 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T11:38:35.069 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-13T11:38:35.080 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-13T11:38:35.117 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T11:38:35.382 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pygments. 2024-09-13T11:38:35.410 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-13T11:38:35.446 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T11:38:35.448 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T11:38:35.475 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-13T11:38:35.513 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-13T11:38:35.790 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T11:38:35.807 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-13T11:38:35.848 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-13T11:38:35.888 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T11:38:35.917 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-13T11:38:35.954 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-13T11:38:36.223 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-toml. 2024-09-13T11:38:36.252 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-13T11:38:36.290 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-13T11:38:36.552 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package radosgw. 2024-09-13T11:38:36.580 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../116-radosgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:36.593 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T11:38:36.618 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:36.621 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-13T11:38:36.659 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T11:38:36.963 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T11:38:36.980 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-13T11:38:37.012 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-13T11:38:37.248 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T11:38:37.265 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-13T11:38:37.298 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-13T11:38:37.721 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T11:38:37.738 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../117-rbd-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:37.779 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:38.060 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-13T11:38:38.252 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package radosgw. 2024-09-13T11:38:38.281 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../116-radosgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:38.309 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T11:38:38.318 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:38.410 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T11:38:38.657 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-13T11:38:38.985 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-13T11:38:39.205 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T11:38:39.221 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../117-rbd-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T11:38:39.263 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:39.431 INFO:teuthology.orchestra.run.smithi005.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-13T11:38:39.559 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-13T11:38:39.748 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:39.801 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T11:38:39.918 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T11:38:40.188 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-13T11:38:40.231 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user cephadm....done 2024-09-13T11:38:40.387 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T11:38:40.532 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-13T11:38:40.689 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-13T11:38:40.945 INFO:teuthology.orchestra.run.smithi039.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-13T11:38:41.049 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-13T11:38:41.175 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-13T11:38:41.288 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:41.459 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T11:38:41.794 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T11:38:41.820 INFO:teuthology.orchestra.run.smithi039.stdout:Adding system user cephadm....done 2024-09-13T11:38:41.920 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-13T11:38:41.969 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T11:38:42.230 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T11:38:42.245 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-13T11:38:42.496 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-13T11:38:42.607 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-13T11:38:42.622 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-13T11:38:42.866 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-13T11:38:42.881 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T11:38:43.134 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-13T11:38:43.191 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T11:38:43.317 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-13T11:38:43.352 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-13T11:38:43.612 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-13T11:38:43.618 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T11:38:43.871 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-13T11:38:43.962 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-13T11:38:44.130 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:44.222 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-13T11:38:44.390 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T11:38:44.498 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-13T11:38:44.516 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T11:38:44.650 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T11:38:44.741 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-13T11:38:44.751 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-13T11:38:44.985 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-13T11:38:44.992 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-13T11:38:45.318 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-13T11:38:45.354 INFO:teuthology.orchestra.run.smithi005.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-13T11:38:45.409 INFO:teuthology.orchestra.run.smithi005.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-13T11:38:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T11:38:45.606 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-13T11:38:45.652 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:45.903 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T11:38:45.974 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T11:38:46.029 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T11:38:46.100 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-13T11:38:46.155 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T11:38:46.281 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-13T11:38:46.360 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T11:38:46.540 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-13T11:38:46.602 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-13T11:38:46.912 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up zip (3.0-12build2) ... 2024-09-13T11:38:46.930 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-13T11:38:46.963 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-13T11:38:47.027 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T11:38:47.038 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T11:38:47.136 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-13T11:38:47.537 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T11:38:47.663 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-13T11:38:47.757 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T11:38:47.939 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T11:38:48.200 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T11:38:48.309 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-13T11:38:48.341 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-13T11:38:48.410 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T11:38:48.667 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up zip (3.0-12build2) ... 2024-09-13T11:38:48.702 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-13T11:38:48.784 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T11:38:49.095 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-13T11:38:49.363 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T11:38:49.494 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T11:38:49.754 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T11:38:49.806 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T11:38:49.880 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-13T11:38:50.006 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T11:38:50.108 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T11:38:50.315 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-13T11:38:50.451 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-13T11:38:50.702 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-13T11:38:50.724 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-13T11:38:50.811 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:50.975 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T11:38:51.129 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T11:38:51.335 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T11:38:51.455 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T11:38:51.629 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T11:38:51.980 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-13T11:38:52.231 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-13T11:38:52.357 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:38:52.691 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T11:38:52.703 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T11:38:52.954 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:38:53.017 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T11:38:53.088 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-13T11:38:53.356 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T11:38:53.482 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-13T11:38:53.633 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T11:38:53.935 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-13T11:38:54.169 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-13T11:38:54.248 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T11:38:54.404 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:38:54.508 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:38:54.529 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-13T11:38:54.633 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-13T11:38:54.865 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-13T11:38:54.885 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T11:38:54.985 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-13T11:38:55.112 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T11:38:55.124 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-13T11:38:55.383 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-13T11:38:55.430 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-13T11:38:55.634 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T11:38:55.706 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-13T11:38:55.860 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-13T11:38:55.965 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:38:56.091 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-13T11:38:56.111 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T11:38:56.237 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-13T11:38:56.417 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-13T11:38:56.530 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T11:38:56.656 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T11:38:56.668 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-13T11:38:56.919 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-13T11:38:56.949 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T11:38:57.187 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T11:38:57.275 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-13T11:38:57.438 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-13T11:38:57.510 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T11:38:57.635 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-13T11:38:57.714 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T11:38:57.761 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T11:38:57.840 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-13T11:38:57.850 INFO:teuthology.orchestra.run.smithi005.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-13T11:38:58.088 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T11:38:58.141 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T11:38:58.214 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T11:38:58.250 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T11:38:58.526 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T11:38:58.607 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-13T11:38:58.811 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-13T11:38:58.875 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-13T11:38:58.976 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-13T11:38:59.087 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T11:38:59.210 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:38:59.213 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-13T11:38:59.337 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T11:38:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T11:38:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-13T11:38:59.654 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T11:38:59.666 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T11:38:59.792 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T11:38:59.964 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T11:39:00.176 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-13T11:39:00.240 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-13T11:39:00.366 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-13T11:39:00.427 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-13T11:39:00.562 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-13T11:39:00.642 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-13T11:39:00.821 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T11:39:00.926 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T11:39:00.947 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T11:39:01.246 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-13T11:39:01.273 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T11:39:01.591 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T11:39:01.631 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T11:39:01.867 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-13T11:39:01.950 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T11:39:01.984 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-13T11:39:02.244 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-13T11:39:02.277 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T11:39:02.403 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:02.511 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T11:39:02.529 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:02.655 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-13T11:39:02.873 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-13T11:39:02.992 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:03.101 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:03.202 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-13T11:39:03.266 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T11:39:03.462 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:03.609 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T11:39:03.755 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:03.881 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:03.954 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T11:39:04.080 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:04.098 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T11:39:04.098 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T11:39:04.206 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:04.332 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-13T11:39:04.599 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:04.643 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:04.733 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:04.769 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:04.876 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:04.895 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-13T11:39:04.994 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T11:39:05.180 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:05.323 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:05.448 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:05.676 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T11:39:05.676 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T11:39:06.116 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:06.217 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:06.297 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:06.318 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:06.423 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:06.427 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T11:39:06.549 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:06.675 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:06.801 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:07.294 INFO:teuthology.orchestra.run.smithi005.stdout:Adding group ceph....done 2024-09-13T11:39:07.671 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user ceph....done 2024-09-13T11:39:07.708 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:07.832 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:07.839 INFO:teuthology.orchestra.run.smithi005.stdout:Setting system user ceph properties....done 2024-09-13T11:39:07.858 INFO:teuthology.orchestra.run.smithi005.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T11:39:07.958 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:08.000 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T11:39:08.059 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:08.160 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:08.318 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T11:39:08.686 INFO:teuthology.orchestra.run.smithi039.stdout:Adding group ceph....done 2024-09-13T11:39:08.786 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:08.928 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:09.071 INFO:teuthology.orchestra.run.smithi039.stdout:Adding system user ceph....done 2024-09-13T11:39:09.239 INFO:teuthology.orchestra.run.smithi039.stdout:Setting system user ceph properties....done 2024-09-13T11:39:09.259 INFO:teuthology.orchestra.run.smithi039.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T11:39:09.404 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T11:39:09.414 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:39:09.414 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:39:09.776 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T11:39:09.911 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:10.236 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:10.274 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T11:39:10.362 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:10.782 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:10.830 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:39:10.831 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:39:11.021 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T11:39:11.021 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T11:39:11.280 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:11.542 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:11.658 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T11:39:11.771 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T11:39:11.771 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T11:39:12.107 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:12.260 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:12.356 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T11:39:12.356 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T11:39:12.544 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T11:39:12.544 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T11:39:12.825 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:13.028 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:13.039 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T11:39:13.040 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T11:39:13.154 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:13.305 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:13.501 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:13.523 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T11:39:13.523 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T11:39:13.779 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T11:39:13.780 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T11:39:14.134 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:14.228 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:14.285 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:14.354 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:14.411 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:14.505 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:14.562 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:14.706 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T11:39:14.706 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T11:39:14.941 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-13T11:39:15.041 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-13T11:39:15.156 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:15.185 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-13T11:39:15.282 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:15.408 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:15.559 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T11:39:15.921 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-13T11:39:16.030 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-13T11:39:16.165 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-13T11:39:16.319 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-13T11:39:16.971 INFO:teuthology.orchestra.run.smithi005.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T11:39:16.977 DEBUG:teuthology.parallel:result is None 2024-09-13T11:39:17.316 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-13T11:39:17.975 INFO:teuthology.orchestra.run.smithi039.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T11:39:17.981 DEBUG:teuthology.parallel:result is None 2024-09-13T11:39:17.981 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:39:18.139 DEBUG:teuthology.orchestra.run.smithi005:> dpkg-query -W -f '${Version}' ceph 2024-09-13T11:39:18.159 INFO:teuthology.orchestra.run.smithi005.stdout:19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:39:18.160 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:39:18.160 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697-gd58f7306-1jammy is installed. 2024-09-13T11:39:18.162 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:39:18.315 DEBUG:teuthology.orchestra.run.smithi039:> dpkg-query -W -f '${Version}' ceph 2024-09-13T11:39:18.337 INFO:teuthology.orchestra.run.smithi039.stdout:19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:39:18.337 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T11:39:18.337 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697-gd58f7306-1jammy is installed. 2024-09-13T11:39:18.339 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T11:39:18.340 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:18.340 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:39:18.355 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:18.355 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:39:18.395 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T11:39:18.395 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:18.395 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:39:18.412 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:39:18.467 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:18.467 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:39:18.481 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:39:18.536 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T11:39:18.537 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:18.537 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:39:18.551 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:39:18.607 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:18.608 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:39:18.621 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:39:18.676 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T11:39:18.677 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:18.677 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:39:18.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:39:18.747 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:18.747 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:39:18.760 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:39:18.817 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T11:39:18.899 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T11:39:18.900 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /var/log/ceph 2024-09-13T11:39:18.902 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /var/log/ceph 2024-09-13T11:39:18.916 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T11:39:18.916 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T11:39:18.957 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T11:39:18.973 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T11:39:18.973 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T11:39:19.012 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T11:39:19.038 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T11:39:19.038 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T11:39:19.038 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428', 'branch': 'wip-aclamk-cbt-combined', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_5925}', 'email': 'akupczyk@redhat.com', 'first_in_suite': False, 'job_id': '7903428', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'owner': 'scheduled_akupczyk@teuthology', 'priority': 160, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 5394, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'sleep_before_teardown': 0, 'subset': '582/12160', 'suite': 'rados', 'suite_branch': 'wip-aclamk-cbt-combined', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'targets': {'smithi005.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDISKxw6WkgLhz0RS4rY2ZEAPyeTs79c0ri10Vt78mSl3dy7EfYqxAu7uwp4LJpcpDKTdIQnFgFAlmrY/7LR1Jc=', 'smithi039.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIIbiBSVAehJs0yx6kwab3IYTXeLbYBX0bMHVgRYoyNJbLdpH8MO1b+709X5587ggt4/22BDGRPvKfVW1TaMP9Q='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1, 'thrash_many': True}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-13_09:37:56', 'tube': 'smithi', 'user': 'akupczyk', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T11:39:19.039 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T11:39:19.069 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T11:39:19.077 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T11:39:19.116 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T11:39:19.133 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:19.133 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T11:39:19.177 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T11:39:19.177 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 2024-09-13T11:39:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T11:39:19.225 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:39:19.225 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 882 Links: 1 2024-09-13T11:39:19.225 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:39:19.225 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-13 11:33:46.058668069 +0000 2024-09-13T11:39:19.225 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-13 11:33:45.810670284 +0000 2024-09-13T11:39:19.225 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-13 11:33:45.810670284 +0000 2024-09-13T11:39:19.225 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-09-13T11:39:19.225 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T11:39:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-13T11:39:19.279 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-13T11:39:19.279 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000310278 s, 1.7 MB/s 2024-09-13T11:39:19.280 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T11:39:19.329 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 2024-09-13T11:39:19.380 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T11:39:19.381 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:39:19.381 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-09-13T11:39:19.381 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:39:19.381 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-13 11:33:46.502664103 +0000 2024-09-13T11:39:19.381 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-13 11:33:46.270666175 +0000 2024-09-13T11:39:19.381 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-13 11:33:46.270666175 +0000 2024-09-13T11:39:19.381 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-09-13T11:39:19.381 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T11:39:19.435 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-13T11:39:19.435 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-13T11:39:19.435 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000309208 s, 1.7 MB/s 2024-09-13T11:39:19.437 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T11:39:19.485 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 2024-09-13T11:39:19.532 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T11:39:19.532 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:39:19.532 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-09-13T11:39:19.532 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:39:19.532 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-13 11:33:46.982659815 +0000 2024-09-13T11:39:19.532 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-13 11:33:46.718662173 +0000 2024-09-13T11:39:19.533 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-13 11:33:46.718662173 +0000 2024-09-13T11:39:19.533 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-09-13T11:39:19.533 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T11:39:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-13T11:39:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-13T11:39:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000255567 s, 2.0 MB/s 2024-09-13T11:39:19.588 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T11:39:19.637 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 2024-09-13T11:39:19.684 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T11:39:19.684 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:39:19.684 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 940 Links: 1 2024-09-13T11:39:19.684 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:39:19.684 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-13 11:33:47.198657885 +0000 2024-09-13T11:39:19.684 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-13 11:33:47.194657921 +0000 2024-09-13T11:39:19.684 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-13 11:33:47.194657921 +0000 2024-09-13T11:39:19.684 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-09-13T11:39:19.685 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T11:39:19.738 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-13T11:39:19.738 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-13T11:39:19.738 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000313794 s, 1.6 MB/s 2024-09-13T11:39:19.739 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T11:39:19.789 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T11:39:19.789 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:19.789 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T11:39:19.795 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T11:39:19.795 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2024-09-13T11:39:19.842 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T11:39:19.842 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:39:19.842 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 863 Links: 1 2024-09-13T11:39:19.842 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:39:19.842 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-13 11:33:46.399936248 +0000 2024-09-13T11:39:19.842 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-13 11:33:46.151889359 +0000 2024-09-13T11:39:19.842 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-13 11:33:46.151889359 +0000 2024-09-13T11:39:19.842 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2024-09-13T11:39:19.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T11:39:19.897 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-13T11:39:19.897 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-13T11:39:19.897 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000534366 s, 958 kB/s 2024-09-13T11:39:19.898 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T11:39:19.947 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2024-09-13T11:39:19.994 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T11:39:19.994 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:39:19.994 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 879 Links: 1 2024-09-13T11:39:19.994 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:39:19.994 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-13 11:33:46.888028513 +0000 2024-09-13T11:39:19.994 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-13 11:33:46.647983136 +0000 2024-09-13T11:39:19.994 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-13 11:33:46.647983136 +0000 2024-09-13T11:39:19.994 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2024-09-13T11:39:19.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T11:39:20.048 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-13T11:39:20.049 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-13T11:39:20.049 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000431805 s, 1.2 MB/s 2024-09-13T11:39:20.050 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T11:39:20.102 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2024-09-13T11:39:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T11:39:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:39:20.154 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-09-13T11:39:20.154 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:39:20.154 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-13 11:33:47.352113791 +0000 2024-09-13T11:39:20.154 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-13 11:33:47.124072280 +0000 2024-09-13T11:39:20.154 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-13 11:33:47.124072280 +0000 2024-09-13T11:39:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2024-09-13T11:39:20.154 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T11:39:20.212 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-13T11:39:20.212 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-13T11:39:20.212 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000499647 s, 1.0 MB/s 2024-09-13T11:39:20.214 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T11:39:20.263 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2024-09-13T11:39:20.309 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T11:39:20.309 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:39:20.310 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-09-13T11:39:20.310 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:39:20.310 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-13 11:33:47.572153846 +0000 2024-09-13T11:39:20.310 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-13 11:33:47.568153118 +0000 2024-09-13T11:39:20.310 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-13 11:33:47.568153118 +0000 2024-09-13T11:39:20.310 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2024-09-13T11:39:20.310 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T11:39:20.364 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-13T11:39:20.364 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-13T11:39:20.364 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000497176 s, 1.0 MB/s 2024-09-13T11:39:20.365 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T11:39:20.415 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T11:39:20.415 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-13T11:39:20.415 INFO:tasks.ceph:Generating config... 2024-09-13T11:39:20.416 INFO:tasks.ceph:[client] debug monc = 20 2024-09-13T11:39:20.416 INFO:tasks.ceph:[client] debug ms = 1 2024-09-13T11:39:20.416 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-13T11:39:20.416 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-13T11:39:20.416 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-13T11:39:20.416 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-13T11:39:20.417 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-13T11:39:20.417 INFO:tasks.ceph:[global] ms type = async 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-13T11:39:20.417 INFO:tasks.ceph:[mon] paxos min = 10 2024-09-13T11:39:20.418 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-13T11:39:20.418 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-09-13T11:39:20.418 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T11:39:20.418 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-13T11:39:20.418 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-13T11:39:20.418 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T11:39:20.418 INFO:tasks.ceph:[osd] bluestore write v2 = False 2024-09-13T11:39:20.418 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-13T11:39:20.418 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-13T11:39:20.418 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-13T11:39:20.418 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T11:39:20.418 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T11:39:20.418 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-13T11:39:20.418 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T11:39:20.419 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T11:39:20.419 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T11:39:20.419 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-13T11:39:20.419 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-13T11:39:20.419 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T11:39:20.419 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-13T11:39:20.419 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-13T11:39:20.419 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T11:39:20.419 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-13T11:39:20.419 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-13T11:39:20.419 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T11:39:20.419 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T11:39:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T11:39:20.478 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T11:39:20.557 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:39:20.611 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.5'), ('mon.c', '[v2:172.21.15.5:3301,v1:172.21.15.5:6790]'), ('mon.b', '172.21.15.39')] 2024-09-13T11:39:20.611 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790],172.21.15.39', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-13T11:39:20.612 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:20.612 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T11:39:20.660 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.5 --addv c '[v2:172.21.15.5:3301,v1:172.21.15.5:6790]' --add b 172.21.15.39 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:39:20.748 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:39:20.749 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: generated fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:20.749 INFO:teuthology.orchestra.run.smithi005.stdout:setting min_mon_release = quincy 2024-09-13T11:39:20.749 INFO:teuthology.orchestra.run.smithi005.stdout:epoch 0 2024-09-13T11:39:20.749 INFO:teuthology.orchestra.run.smithi005.stdout:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:20.749 INFO:teuthology.orchestra.run.smithi005.stdout:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:20.750 INFO:teuthology.orchestra.run.smithi005.stdout:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:20.750 INFO:teuthology.orchestra.run.smithi005.stdout:min_mon_release 17 (quincy) 2024-09-13T11:39:20.750 INFO:teuthology.orchestra.run.smithi005.stdout:election_strategy: 1 2024-09-13T11:39:20.750 INFO:teuthology.orchestra.run.smithi005.stdout:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:20.750 INFO:teuthology.orchestra.run.smithi005.stdout:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:20.751 INFO:teuthology.orchestra.run.smithi005.stdout:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:20.751 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-13T11:39:20.753 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T11:39:20.758 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 59046471-79a9-4d90-9616-da0b446053a7... 2024-09-13T11:39:20.759 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T11:39:20.800 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T11:39:20.827 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2024-09-13T11:39:20.827 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2024-09-13T11:39:20.827 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T11:39:20.827 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2024-09-13T11:39:20.827 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.827 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T11:39:20.828 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2024-09-13T11:39:20.829 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790],172.21.15.39" 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: mon client directed command retry = 5 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: mon election default strategy = 1 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: ms bind msgr1 = False 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: ms bind msgr2 = True 2024-09-13T11:39:20.830 INFO:teuthology.orchestra.run.smithi005.stdout: ms inject socket failures = 5000 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: ms type = async 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: osd mclock profile = high_recovery_ops 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.831 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug trim objects = true 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2024-09-13T11:39:20.832 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore compression algorithm = lz4 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore compression mode = aggressive 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore write v2 = False 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore zero block detection = True 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-13T11:39:20.833 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 20 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify cached snaps = True 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify missing on start = True 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout: osd mclock override recovery settings = True 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout: osd op queue = debug_random 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout: osd op queue cut off = debug_random 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.834 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2024-09-13T11:39:20.835 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: # 1m isn't quite enough 2024-09-13T11:39:20.836 INFO:teuthology.orchestra.run.smithi005.stdout: mon_down_mkfs_grace = 2m 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_filestore_osds = false 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: mon min osdmap epochs = 25 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: mon osdmap full prune interval = 2 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: mon osdmap full prune min = 15 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: mon osdmap full prune txsize = 2 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: mon scrub inject crc mismatch = 0.01 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: mon scrub inject missing keys = 0.05 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: mon scrub interval = 300 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: paxos min = 10 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: paxos service trim min = 5 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: paxos trim min = 10 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2024-09-13T11:39:20.837 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2024-09-13T11:39:20.838 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2024-09-13T11:39:20.838 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2024-09-13T11:39:20.838 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T11:39:20.838 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T11:39:20.838 INFO:teuthology.orchestra.run.smithi005.stdout: debug monc = 20 2024-09-13T11:39:20.838 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-13T11:39:20.838 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2024-09-13T11:39:20.838 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2024-09-13T11:39:20.838 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.839 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-09-13T11:39:20.840 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.841 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790],172.21.15.39" 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: mon client directed command retry = 5 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: mon election default strategy = 1 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: ms bind msgr1 = False 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: ms bind msgr2 = True 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: ms inject socket failures = 5000 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: ms type = async 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-09-13T11:39:20.842 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-09-13T11:39:20.843 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore compression algorithm = lz4 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore compression mode = aggressive 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-09-13T11:39:20.844 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore write v2 = False 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore zero block detection = True 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 20 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 20 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 10 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify cached snaps = True 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify missing on start = True 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock override recovery settings = True 2024-09-13T11:39:20.845 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: osd op queue = debug_random 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: osd op queue cut off = debug_random 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-13T11:39:20.846 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.847 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: mon min osdmap epochs = 25 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune interval = 2 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune min = 15 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune txsize = 2 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: mon scrub inject crc mismatch = 0.01 2024-09-13T11:39:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: mon scrub inject missing keys = 0.05 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: mon scrub interval = 300 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: paxos min = 10 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: paxos service trim min = 5 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: paxos trim min = 10 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: debug monc = 20 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-13T11:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-09-13T11:39:20.850 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-09-13T11:39:20.850 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-09-13T11:39:20.851 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T11:39:20.851 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T11:39:20.941 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T11:39:20.941 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:20.941 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T11:39:20.946 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:20.946 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T11:39:20.992 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T11:39:20.992 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:20.992 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T11:39:20.992 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:39:21.057 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:21.057 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:39:21.104 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T11:39:21.104 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:21.104 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T11:39:21.104 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:39:21.129 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:21.129 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:39:21.177 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T11:39:21.177 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T11:39:21.178 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T11:39:21.264 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T11:39:21.279 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T11:39:21.279 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T11:39:21.279 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T11:39:21.355 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T11:39:21.368 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T11:39:21.368 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-13T11:39:21.369 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T11:39:21.383 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T11:39:21.383 INFO:tasks.ceph:role: osd.0 2024-09-13T11:39:21.383 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T11:39:21.383 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T11:39:21.440 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:39:21.440 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:39:21.440 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:39:21.440 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:39:21.440 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:39:21.440 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:39:21.440 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:39:21.440 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:39:21.440 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:39:21.441 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:39:21.464 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-09-13T11:39:21.466 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-13T11:39:21.466 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T11:39:21.524 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T11:39:21.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:39:21.577 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:39:21.578 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T11:39:21.630 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T11:39:21.631 INFO:tasks.ceph:role: osd.1 2024-09-13T11:39:21.631 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T11:39:21.631 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T11:39:21.685 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:39:21.685 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:39:21.685 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:39:21.685 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:39:21.685 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:39:21.685 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:39:21.685 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:39:21.685 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:39:21.685 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:39:21.686 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:39:21.709 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-09-13T11:39:21.711 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-13T11:39:21.711 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-13T11:39:21.771 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T11:39:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:39:21.825 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:39:21.825 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T11:39:21.881 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T11:39:21.881 INFO:tasks.ceph:role: osd.2 2024-09-13T11:39:21.881 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T11:39:21.882 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T11:39:21.938 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:39:21.938 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:39:21.938 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:39:21.938 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:39:21.938 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:39:21.938 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:39:21.938 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:39:21.938 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:39:21.938 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:39:21.939 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:39:21.961 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-09-13T11:39:21.963 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-13T11:39:21.963 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-13T11:39:22.023 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T11:39:22.081 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:39:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:39:22.083 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:39:22.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:22.200+0000 7fc757e127c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T11:39:22.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:22.200+0000 7fc757e127c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T11:39:22.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:22.200+0000 7fc757e127c0 -1 bdev(0x5614f4227800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-13T11:39:22.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:22.200+0000 7fc757e127c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T11:39:25.333 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T11:39:25.347 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:39:25.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:25.432+0000 7fc5e26b17c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T11:39:25.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:25.432+0000 7fc5e26b17c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T11:39:25.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:25.432+0000 7fc5e26b17c0 -1 bdev(0x55e2ba015800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-13T11:39:25.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:25.432+0000 7fc5e26b17c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T11:39:28.548 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T11:39:28.605 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:39:28.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:28.688+0000 7ff06557c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T11:39:28.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:28.688+0000 7ff06557c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T11:39:28.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:28.688+0000 7ff06557c7c0 -1 bdev(0x55c3091c9800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-13T11:39:28.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:28.688+0000 7ff06557c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T11:39:31.827 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T11:39:31.884 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-13T11:39:31.899 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T11:39:31.899 INFO:tasks.ceph:role: osd.3 2024-09-13T11:39:31.899 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T11:39:31.899 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T11:39:31.983 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:39:31.983 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:39:31.983 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:39:31.983 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:39:31.984 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:39:31.984 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:39:31.984 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:39:31.984 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:39:31.984 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:39:31.984 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:39:33.269 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-13T11:39:33.271 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-13T11:39:33.271 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-13T11:39:33.331 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-13T11:39:33.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:39:33.382 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:39:33.382 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-13T11:39:33.437 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T11:39:33.437 INFO:tasks.ceph:role: osd.4 2024-09-13T11:39:33.437 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T11:39:33.437 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T11:39:33.522 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:39:33.523 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:39:33.523 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:39:33.523 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:39:33.523 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:39:33.523 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:39:33.523 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:39:33.523 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:39:33.523 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:39:33.523 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:39:34.813 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-13T11:39:34.815 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-13T11:39:34.815 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-13T11:39:34.877 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-13T11:39:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:39:34.930 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:39:34.930 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-13T11:39:34.984 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T11:39:34.984 INFO:tasks.ceph:role: osd.5 2024-09-13T11:39:34.984 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T11:39:34.984 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T11:39:35.069 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:39:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:39:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:39:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-13T11:39:35.070 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:39:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:39:35.070 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:39:35.070 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T11:39:35.070 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:39:35.070 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:39:36.359 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-13T11:39:36.361 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-13T11:39:36.362 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-13T11:39:36.425 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-13T11:39:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:39:36.481 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T11:39:36.482 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:39:36.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:36.598+0000 7f1e7b1357c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-13T11:39:36.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:36.602+0000 7f1e7b1357c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-13T11:39:36.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:36.602+0000 7f1e7b1357c0 -1 bdev(0x560f05313800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-13T11:39:36.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:36.602+0000 7f1e7b1357c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-13T11:39:39.782 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T11:39:39.797 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:39:39.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:39.878+0000 7f71b1ce87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-13T11:39:39.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:39.882+0000 7f71b1ce87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-13T11:39:39.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:39.882+0000 7f71b1ce87c0 -1 bdev(0x558cd05c1800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-13T11:39:39.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:39.882+0000 7f71b1ce87c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-13T11:39:43.020 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T11:39:43.078 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:39:43.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:43.162+0000 7f16875827c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-13T11:39:43.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:43.162+0000 7f16875827c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-13T11:39:43.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:43.162+0000 7f16875827c0 -1 bdev(0x55631a2d1800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-13T11:39:43.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:43.162+0000 7f16875827c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-13T11:39:46.301 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-13T11:39:46.358 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T11:39:46.359 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:46.359 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T11:39:46.373 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:46.373 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T11:39:46.427 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:46.428 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T11:39:46.483 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:46.483 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T11:39:46.497 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:46.497 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-13T11:39:46.553 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:46.554 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-13T11:39:46.613 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:46.613 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-13T11:39:46.668 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:46.668 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T11:39:46.717 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T11:39:46.718 DEBUG:teuthology.orchestra.run.smithi005:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T11:39:46.720 DEBUG:teuthology.orchestra.run.smithi039:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T11:39:46.763 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.0] 2024-09-13T11:39:46.763 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBqJORm6vT9CxAAmkwT7Ui71JVpE0oY3EANMQ== 2024-09-13T11:39:46.763 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.1] 2024-09-13T11:39:46.763 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBtJORmbNnNGRAAMd+6tTR4BXwXH5remzFspw== 2024-09-13T11:39:46.763 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.2] 2024-09-13T11:39:46.763 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBwJORmtb4hKRAAq7KawClZW88Q0EP61Fl6ZA== 2024-09-13T11:39:46.764 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.x] 2024-09-13T11:39:46.764 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBpJORma9vUDxAAhm5mgyzkca9teTwRZsQUNg== 2024-09-13T11:39:46.764 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.3] 2024-09-13T11:39:46.764 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQB4JORmONPfIxAAt7mYv9CJcN+pFc1tHTlcIg== 2024-09-13T11:39:46.764 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.4] 2024-09-13T11:39:46.764 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQB7JORmTCWUNBAA55l/GRGktEX0ooNy2G16ng== 2024-09-13T11:39:46.764 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.5] 2024-09-13T11:39:46.764 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQB/JORmX6u7CRAAGnitgDyfmZbHw6X7tWbZow== 2024-09-13T11:39:46.764 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2024-09-13T11:39:46.764 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBpJORmk1NLFRAAiYc5hUL8TdAgaOwMzvAwDA== 2024-09-13T11:39:46.772 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.0] 2024-09-13T11:39:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBqJORm6vT9CxAAmkwT7Ui71JVpE0oY3EANMQ== 2024-09-13T11:39:46.773 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.1] 2024-09-13T11:39:46.773 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBtJORmbNnNGRAAMd+6tTR4BXwXH5remzFspw== 2024-09-13T11:39:46.774 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.2] 2024-09-13T11:39:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBwJORmtb4hKRAAq7KawClZW88Q0EP61Fl6ZA== 2024-09-13T11:39:46.774 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.x] 2024-09-13T11:39:46.774 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBpJORma9vUDxAAhm5mgyzkca9teTwRZsQUNg== 2024-09-13T11:39:46.774 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.3] 2024-09-13T11:39:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB4JORmONPfIxAAt7mYv9CJcN+pFc1tHTlcIg== 2024-09-13T11:39:46.775 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.4] 2024-09-13T11:39:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB7JORmTCWUNBAA55l/GRGktEX0ooNy2G16ng== 2024-09-13T11:39:46.775 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.5] 2024-09-13T11:39:46.775 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB/JORmX6u7CRAAGnitgDyfmZbHw6X7tWbZow== 2024-09-13T11:39:46.775 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2024-09-13T11:39:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBpJORmk1NLFRAAiYc5hUL8TdAgaOwMzvAwDA== 2024-09-13T11:39:46.777 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:39:46.813 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:39:46.868 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:39:46.908 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:39:46.952 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:39:46.997 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:39:47.044 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:39:47.084 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:39:47.131 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:39:47.172 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:39:47.215 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:39:47.256 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:39:47.300 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:39:47.303 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:39:47.388 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T11:39:47.390 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T11:39:47.479 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T11:39:47.479 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T11:39:47.491 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T11:39:47.591 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T11:39:47.603 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-13T11:39:47.662 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T11:39:47.753 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-13T11:39:47.764 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-13T11:39:47.778 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T11:39:47.887 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-13T11:39:47.899 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:39:47.902 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:39:47.944 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T11:39:47.945 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T11:39:47.945 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T11:39:47.948 INFO:tasks.ceph.mon.a:Started 2024-09-13T11:39:47.949 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T11:39:47.949 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T11:39:47.951 INFO:tasks.ceph.mon.c:Started 2024-09-13T11:39:47.951 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T11:39:47.952 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T11:39:47.990 INFO:tasks.ceph.mon.b:Started 2024-09-13T11:39:47.990 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T11:39:47.990 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T11:39:47.991 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T11:39:47.993 INFO:tasks.ceph.mgr.x:Started 2024-09-13T11:39:47.993 DEBUG:tasks.ceph:set 0 configs 2024-09-13T11:39:47.993 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T11:39:47.993 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T11:39:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.112+0000 7fa55755a640 10 monclient: get_monmap_and_config 2024-09-13T11:39:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.112+0000 7fa55755a640 10 monclient: build_initial_monmap 2024-09-13T11:39:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 10 monclient: monmap: 2024-09-13T11:39:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:39:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:48.116833+0000 2024-09-13T11:39:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:48.116833+0000 2024-09-13T11:39:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:39:48.116 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:39:48.116 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:39:48.116 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:39:48.116 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:39:48.116 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:48.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 1 Processor -- start 2024-09-13T11:39:48.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 1 -- start start 2024-09-13T11:39:48.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 10 monclient: init 2024-09-13T11:39:48.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:48.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-13T11:39:48.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:48.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 10 monclient(hunting): picked mon.noname-a con 0x7fa550134a70 addr v2:172.21.15.5:3300/0 2024-09-13T11:39:48.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa55009ee00 addr v1:172.21.15.5:6789/0 2024-09-13T11:39:48.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:48.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 10 monclient(hunting): picked mon.noname-c con 0x7fa55009c380 addr v2:172.21.15.39:3300/0 2024-09-13T11:39:48.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa55009e420 con 0x7fa55009ee00 2024-09-13T11:39:48.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:48.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa550135390 con 0x7fa550134a70 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fa55009e140 con 0x7fa55009c380 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa55755a640 10 monclient(hunting): authenticate will time out at 991.465203s 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa554af5640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa55009ee00 0x7fa55009bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:53364/0 (socket says 172.21.15.5:53364) 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa5552f6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa554af5640 1 -- 172.21.15.5:0/1224044626 learned_addr learned my addr 172.21.15.5:0/1224044626 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa5552f6640 10 monclient(hunting): get_auth_request con 0x7fa550134a70 auth_method 0 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa5552f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa5552f6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa5552f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 msgr2=0x7fa550134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:39:48.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 msgr2=0x7fa550134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:39:48.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:48.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa555af7640 10 monclient(hunting): get_auth_request con 0x7fa55009c380 auth_method 0 2024-09-13T11:39:48.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa555af7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:48.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa555af7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:48.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa555af7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:48.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:39:48.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa555af7640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 msgr2=0x7fa55013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:39:48.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa555af7640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 msgr2=0x7fa55013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:39:48.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:48.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.116+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:39:48.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:48.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa5552f6640 10 monclient(hunting): get_auth_request con 0x7fa550134a70 auth_method 0 2024-09-13T11:39:48.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa5552f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:48.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa5552f6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:48.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa5552f6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:48.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:48.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa555af7640 10 monclient(hunting): get_auth_request con 0x7fa55009c380 auth_method 0 2024-09-13T11:39:48.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa555af7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:48.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa555af7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:48.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa555af7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:48.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 msgr2=0x7fa550134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:39:48.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 msgr2=0x7fa550134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:39:48.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:48.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:39:48.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa555af7640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 msgr2=0x7fa55013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:39:48.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa555af7640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 msgr2=0x7fa55013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:39:48.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:48.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.320+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:39:48.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa5552f6640 10 monclient(hunting): get_auth_request con 0x7fa550134a70 auth_method 0 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa5552f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa5552f6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa5552f6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 msgr2=0x7fa550134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 msgr2=0x7fa550134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa555af7640 10 monclient(hunting): get_auth_request con 0x7fa55009c380 auth_method 0 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa555af7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa555af7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa555af7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:48.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:39:48.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa555af7640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 msgr2=0x7fa55013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:39:48.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa555af7640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 msgr2=0x7fa55013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:39:48.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:48.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:48.724+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:39:49.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:49.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa5552f6640 10 monclient(hunting): get_auth_request con 0x7fa550134a70 auth_method 0 2024-09-13T11:39:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa5552f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa5552f6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa5552f6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 msgr2=0x7fa550134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:39:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 msgr2=0x7fa550134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:39:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:39:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa555af7640 10 monclient(hunting): get_auth_request con 0x7fa55009c380 auth_method 0 2024-09-13T11:39:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa555af7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa555af7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa555af7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:49.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa555af7640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 msgr2=0x7fa55013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:39:49.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa555af7640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 msgr2=0x7fa55013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:39:49.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:49.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:49.528+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:39:51.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 10 monclient(hunting): tick 2024-09-13T11:39:51.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 10 monclient(hunting): _check_auth_tickets 2024-09-13T11:39:51.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 1 monclient(hunting): continuing hunt 2024-09-13T11:39:51.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 10 monclient(hunting): _reopen_session rank -1 2024-09-13T11:39:51.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 msgr2=0x7fa55013c810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:39:51.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.39:3300/0 conn(0x7fa55009c380 0x7fa55013c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 1 -- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 msgr2=0x7fa550134e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 1 --2- 172.21.15.5:0/1224044626 >> v2:172.21.15.5:3300/0 conn(0x7fa550134a70 0x7fa550134e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 1 -- 172.21.15.5:0/1224044626 >> v1:172.21.15.5:6789/0 conn(0x7fa55009ee00 legacy=0x7fa55009bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 10 monclient: _add_conns ranks=[2,1] 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 10 monclient(hunting): picked mon.noname-b con 0x7fa53c0036c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa53c006200 addr v1:172.21.15.39:6789/0 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 1 -- 172.21.15.5:0/1224044626 --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa55009e700 con 0x7fa53c006200 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 1 -- 172.21.15.5:0/1224044626 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa550135390 con 0x7fa53c0036c0 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa54effd640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa5552f6640 10 monclient(hunting): get_auth_request con 0x7fa53c0036c0 auth_method 0 2024-09-13T11:39:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa5552f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:51.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa5552f6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:51.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa5552f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:51.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 msgr2=0x7fa53c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:39:51.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 msgr2=0x7fa53c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:39:51.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:51.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.116+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:39:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.320+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.320+0000 7fa5552f6640 10 monclient(hunting): get_auth_request con 0x7fa53c0036c0 auth_method 0 2024-09-13T11:39:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.320+0000 7fa5552f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.320+0000 7fa5552f6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.320+0000 7fa5552f6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.320+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 msgr2=0x7fa53c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:39:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.320+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 msgr2=0x7fa53c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:39:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.320+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.320+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:39:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.724+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.724+0000 7fa5552f6640 10 monclient(hunting): get_auth_request con 0x7fa53c0036c0 auth_method 0 2024-09-13T11:39:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.724+0000 7fa5552f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.724+0000 7fa5552f6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.724+0000 7fa5552f6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.724+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 msgr2=0x7fa53c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:39:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.724+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 msgr2=0x7fa53c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:39:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.724+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:51.724+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:39:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:52.528+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:52.528+0000 7fa5552f6640 10 monclient(hunting): get_auth_request con 0x7fa53c0036c0 auth_method 0 2024-09-13T11:39:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:52.528+0000 7fa5552f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:52.528+0000 7fa5552f6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:52.528+0000 7fa5552f6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:52.528+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 msgr2=0x7fa53c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:39:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:52.528+0000 7fa5552f6640 1 -- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 msgr2=0x7fa53c005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:39:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:52.528+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:52.528+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:39:53.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.348+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895110757 0 0) 0x7fa55009e700 con 0x7fa53c006200 2024-09-13T11:39:53.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.348+0000 7fa54ffff640 10 monclient(hunting): _negotiate 2024-09-13T11:39:53.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.348+0000 7fa54ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.348+0000 7fa54ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:53.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.348+0000 7fa54ffff640 10 monclient(hunting): my global_id is 4102 2024-09-13T11:39:53.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.348+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa538003530 con 0x7fa53c006200 2024-09-13T11:39:53.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 <== mon.1 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2322074273 0 0) 0x7fa538003530 con 0x7fa53c006200 2024-09-13T11:39:53.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 msgr2=0x7fa53c005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:39:53.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c0036c0 0x7fa53c005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient: _finish_hunting 0 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 monclient: found mon.noname-c-legacy 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.39:6789/0 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53c00b050 con 0x7fa53c006200 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient: _finish_auth 0 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 <== mon.1 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3199362888 0 0) 0x7fa5440017c0 con 0x7fa53c006200 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa55755a640 5 monclient: authenticate success, global_id 4102 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa55755a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:53.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient: dump: 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] but i'm connected to v1:172.21.15.39:6789/0, reconnecting 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:53.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 >> v1:172.21.15.39:6789/0 conn(0x7fa53c006200 legacy=0x7fa53c0086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:53.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 1 -- 172.21.15.5:0/1224044626 reap_dead start 2024-09-13T11:39:53.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:39:53.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538003a60 0x7fa538003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient(hunting): picked mon.b con 0x7fa538003a60 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:39:53.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 0x7fa538004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient(hunting): picked mon.a con 0x7fa538004810 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:39:53.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa5380053d0 0x7fa53800b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient(hunting): picked mon.c con 0x7fa5380053d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:53.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa550135390 con 0x7fa538004810 2024-09-13T11:39:53.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa53800c2d0 con 0x7fa538003a60 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa53800c4d0 con 0x7fa5380053d0 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa54ffff640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa55755a640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa5552f6640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538003a60 0x7fa538003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa554af5640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 0x7fa538004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa5380053d0 0x7fa53800b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa5552f6640 10 monclient(hunting): get_auth_request con 0x7fa538003a60 auth_method 0 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa5552f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa5552f6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa5552f6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:53.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 10 monclient(hunting): get_auth_request con 0x7fa5380053d0 auth_method 0 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa554af5640 10 monclient(hunting): get_auth_request con 0x7fa538004810 auth_method 0 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa554af5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa554af5640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa554af5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa5552f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa5552f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa5552f6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa554af5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:53.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa554af5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa554af5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 10 monclient(hunting): handle_auth_done global_id 4102 payload 293 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 1 -- 172.21.15.5:0/1224044626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538003a60 msgr2=0x7fa538003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538003a60 0x7fa538003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 1 -- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 msgr2=0x7fa538004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 0x7fa538004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 10 monclient: _finish_hunting 0 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 1 monclient: found mon.c 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 1 -- 172.21.15.5:0/1224044626 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa538003530 con 0x7fa5380053d0 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 10 monclient: _finish_auth 0 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 10 monclient: _check_auth_tickets 2024-09-13T11:39:53.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.356+0000 7fa555af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:53.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa555af7640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa5380053d0 0x7fa53800b9f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa55006f130 tx=0x7fa54001d090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ebc27d4aac6ea58 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5400ab070 con 0x7fa5380053d0 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa54ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa54ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa54ffff640 10 monclient: dump: 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:53.365 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5400a5d90 con 0x7fa5380053d0 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa54ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 10 monclient: get_monmap_and_config success 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 4 set_mon_vals no callback set 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa54ffff640 1 -- 172.21.15.5:0/1224044626 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540003dc0 con 0x7fa5380053d0 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa54ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa54ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa54ffff640 10 monclient: dump: 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:53.367 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:53.367 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:53.367 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:53.367 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:53.367 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:53.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 10 monclient: shutdown 2024-09-13T11:39:53.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:53.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 1 -- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa5380053d0 msgr2=0x7fa53800b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:53.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa5380053d0 0x7fa53800b9f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa55006f130 tx=0x7fa54001d090 comp rx=0 tx=0).stop 2024-09-13T11:39:53.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 1 -- 172.21.15.5:0/1224044626 shutdown_connections 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa5380053d0 0x7fa53800b9f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 0x7fa538004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 1 --2- 172.21.15.5:0/1224044626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538003a60 0x7fa538003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 1 -- 172.21.15.5:0/1224044626 >> 172.21.15.5:0/1224044626 conn(0x7fa5500a61e0 msgr2=0x7fa5500a65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 1 -- 172.21.15.5:0/1224044626 shutdown_connections 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.364+0000 7fa55755a640 1 -- 172.21.15.5:0/1224044626 wait complete. 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient: build_initial_monmap 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient: monmap: 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:39:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:53.370 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:53.370 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:39:53.370 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:39:53.370 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:39:53.370 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:53.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 1 Processor -- start 2024-09-13T11:39:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 1 -- start start 2024-09-13T11:39:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient: init 2024-09-13T11:39:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:39:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa538003a60 0x7fa55013b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient(hunting): picked mon.noname-c con 0x7fa538003a60 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 0x7fa55013bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient(hunting): picked mon.noname-a con 0x7fa538004810 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5380053d0 0x7fa55013c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient(hunting): picked mon.noname-b con 0x7fa5380053d0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa55009bc70 con 0x7fa538004810 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa55009f3f0 con 0x7fa5380053d0 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa55009bdf0 con 0x7fa538003a60 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa55755a640 10 monclient(hunting): authenticate will time out at 996.719174s 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa5552f6640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa538003a60 0x7fa55013b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa5552f6640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa538003a60 0x7fa55013b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:42942/0 (socket says 172.21.15.5:42942) 2024-09-13T11:39:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa5552f6640 1 -- 172.21.15.5:0/3121959596 learned_addr learned my addr 172.21.15.5:0/3121959596 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa5552f6640 10 monclient(hunting): get_auth_request con 0x7fa538003a60 auth_method 0 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa5552f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa5552f6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa5552f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 0x7fa55013bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 10 monclient(hunting): get_auth_request con 0x7fa538004810 auth_method 0 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa5552f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa5552f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa5552f6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:53.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa555af7640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5380053d0 0x7fa55013c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa555af7640 10 monclient(hunting): get_auth_request con 0x7fa5380053d0 auth_method 0 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa555af7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa555af7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa555af7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa555af7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa555af7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa555af7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 10 monclient(hunting): handle_auth_done global_id 4101 payload 824 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 1 -- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa538003a60 msgr2=0x7fa55013b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa538003a60 0x7fa55013b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 1 -- 172.21.15.5:0/3121959596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5380053d0 msgr2=0x7fa55013c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5380053d0 0x7fa55013c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 10 monclient: _finish_hunting 0 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 1 monclient: found mon.noname-a 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.368+0000 7fa554af5640 1 -- 172.21.15.5:0/3121959596 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55011d830 con 0x7fa538004810 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 0x7fa55013bd30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa544004f50 tx=0x7fa5440074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54400f730 con 0x7fa538004810 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: dump: 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.380 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: _finish_auth 0 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa544018070 con 0x7fa538004810 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54400fb30 con 0x7fa538004810 2024-09-13T11:39:53.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa55755a640 5 monclient: authenticate success, global_id 4101 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa55755a640 10 monclient: _renew_subs 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa55755a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa55755a640 1 -- 172.21.15.5:0/3121959596 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa55009d0d0 con 0x7fa538004810 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54f7fe640 4 set_mon_vals no callback set 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: dump: 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:53.382 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:53.383 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:53.383 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:53.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa55755a640 10 monclient: _renew_subs 2024-09-13T11:39:53.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa55755a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:39:53.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa55755a640 1 -- 172.21.15.5:0/3121959596 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa550133750 con 0x7fa538004810 2024-09-13T11:39:53.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 1 -- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 msgr2=0x7fa55013bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-13T11:39:53.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 1 -- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 msgr2=0x7fa55013bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:39:53.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 0x7fa55013bd30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa544004f50 tx=0x7fa5440074f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:39:53.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 0x7fa55013bd30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa544004f50 tx=0x7fa5440074f0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:39:53.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 msgr2=0x7fa55013bd30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa538004810 0x7fa55013bd30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c005420 0x7fa53c005800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient(hunting): picked mon.a con 0x7fa53c005420 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5380053d0 0x7fa53c0084e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient(hunting): picked mon.b con 0x7fa5380053d0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa538003a60 0x7fa53c00a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient(hunting): picked mon.c con 0x7fa538003a60 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c00aac0 con 0x7fa53c005420 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c00ac70 con 0x7fa5380053d0 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa53c008e50 con 0x7fa538003a60 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa55755a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa55755a640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa538003a60 0x7fa53c00a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient(hunting): get_auth_request con 0x7fa538003a60 auth_method 0 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5380053d0 0x7fa53c0084e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa5552f6640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c005420 0x7fa53c005800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:53.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 10 monclient(hunting): get_auth_request con 0x7fa5380053d0 auth_method 0 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa5552f6640 10 monclient(hunting): get_auth_request con 0x7fa53c005420 auth_method 0 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa5552f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa5552f6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa5552f6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:53.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa554af5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa5552f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa5552f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa5552f6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient(hunting): handle_auth_done global_id 4101 payload 293 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 1 -- 172.21.15.5:0/3121959596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5380053d0 msgr2=0x7fa53c0084e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5380053d0 0x7fa53c0084e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 1 -- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c005420 msgr2=0x7fa53c005800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c005420 0x7fa53c005800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient: _finish_hunting 0 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 1 monclient: found mon.c 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 1 -- 172.21.15.5:0/3121959596 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fa53c009120 con 0x7fa538003a60 2024-09-13T11:39:53.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 1 -- 172.21.15.5:0/3121959596 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa55009e140 con 0x7fa538003a60 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 1 -- 172.21.15.5:0/3121959596 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa538003530 con 0x7fa538003a60 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient: _finish_auth 0 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 10 monclient: _check_auth_tickets 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa555af7640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa538003a60 0x7fa53c00a0d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa540003ea0 tx=0x7fa544001e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5400ab040 con 0x7fa538003a60 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: dump: 2024-09-13T11:39:53.388 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5400af400 con 0x7fa538003a60 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa5400af980 con 0x7fa538003a60 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54effd640 4 set_mon_vals no callback set 2024-09-13T11:39:53.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.372+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 <== mon.2 v2:172.21.15.5:3301/0 4 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa5400c2cd0 con 0x7fa538003a60 2024-09-13T11:39:53.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.384+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 <== mon.2 v2:172.21.15.5:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fa540016e70 con 0x7fa538003a60 2024-09-13T11:39:53.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.384+0000 7fa54dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:53.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.384+0000 7fa54dffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:39:53.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.388+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fa5400afc70 con 0x7fa538003a60 2024-09-13T11:39:53.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.388+0000 7fa54dffb640 10 monclient: handle_mon_command_ack 1 not found 2024-09-13T11:39:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.452+0000 7fa55755a640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-13T11:39:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.452+0000 7fa55755a640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-13T11:39:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.452+0000 7fa55755a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.452+0000 7fa55755a640 1 -- 172.21.15.5:0/3121959596 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fa550074680 con 0x7fa538003a60 2024-09-13T11:39:53.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa54dffb640 1 -- 172.21.15.5:0/3121959596 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fa5400c23f0 con 0x7fa538003a60 2024-09-13T11:39:53.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa54dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-13T11:39:53.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa54dffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-13T11:39:53.459 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2024-09-13T11:39:53.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa55755a640 10 monclient: shutdown 2024-09-13T11:39:53.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa55755a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:53.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa55755a640 1 -- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa538003a60 msgr2=0x7fa53c00a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:53.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa55755a640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa538003a60 0x7fa53c00a0d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa540003ea0 tx=0x7fa544001e60 comp rx=0 tx=0).stop 2024-09-13T11:39:53.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa55755a640 1 -- 172.21.15.5:0/3121959596 shutdown_connections 2024-09-13T11:39:53.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa55755a640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa538003a60 0x7fa53c00a0d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa55755a640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa5380053d0 0x7fa53c0084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa55755a640 1 --2- 172.21.15.5:0/3121959596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c005420 0x7fa53c005800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa55755a640 1 -- 172.21.15.5:0/3121959596 >> 172.21.15.5:0/3121959596 conn(0x7fa5500a61e0 msgr2=0x7fa5500a49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:53.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa55755a640 1 -- 172.21.15.5:0/3121959596 shutdown_connections 2024-09-13T11:39:53.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:53.460+0000 7fa55755a640 1 -- 172.21.15.5:0/3121959596 wait complete. 2024-09-13T11:39:53.479 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T11:39:53.479 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T11:39:53.479 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:53.479 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T11:39:53.492 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:53.492 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T11:39:53.546 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:39:53.547 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T11:39:53.606 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:53.606 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-13T11:39:53.620 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:53.621 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-13T11:39:53.677 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:39:53.677 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-13T11:39:53.733 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new 0803d65c-46ec-49b5-a45e-46bcf59ae3f3 0 2024-09-13T11:39:53.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 10 monclient: get_monmap_and_config 2024-09-13T11:39:53.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 10 monclient: build_initial_monmap 2024-09-13T11:39:53.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 10 monclient: monmap: 2024-09-13T11:39:53.903 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 0 2024-09-13T11:39:53.903 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:53.903 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:53.904265+0000 2024-09-13T11:39:53.903 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:53.904265+0000 2024-09-13T11:39:53.903 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:53.903 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:53.903 INFO:teuthology.orchestra.run.smithi039.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:39:53.903 INFO:teuthology.orchestra.run.smithi039.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:39:53.903 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:39:53.903 INFO:teuthology.orchestra.run.smithi039.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:39:53.903 INFO:teuthology.orchestra.run.smithi039.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:39:53.903 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:53.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 1 Processor -- start 2024-09-13T11:39:53.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 1 -- start start 2024-09-13T11:39:53.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 10 monclient: init 2024-09-13T11:39:53.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:53.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa6d8134a70 0x7fa6d8134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d8134a70 addr v2:172.21.15.39:3300/0 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6d809d250 addr v1:172.21.15.5:6789/0 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6d806da20 addr v1:172.21.15.39:6789/0 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6d809c870 con 0x7fa6d809d250 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6d80a2ca0 con 0x7fa6d806da20 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fa6d809c340 con 0x7fa6d8134a70 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6df802640 10 monclient(hunting): authenticate will time out at 960.443305s 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6dd59e640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa6d8134a70 0x7fa6d8134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6dd59e640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa6d8134a70 0x7fa6d8134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50618/0 (socket says 172.21.15.39:50618) 2024-09-13T11:39:53.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6dd59e640 1 -- 172.21.15.39:0/1081295094 learned_addr learned my addr 172.21.15.39:0/1081295094 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6dd59e640 10 monclient(hunting): get_auth_request con 0x7fa6d8134a70 auth_method 0 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6dd59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6dd59e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6dd59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6dd59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6dd59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.902+0000 7fa6dd59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6dd59e640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6dd59e640 1 -- 172.21.15.39:0/1081295094 >> v1:172.21.15.39:6789/0 conn(0x7fa6d806da20 legacy=0x7fa6d813fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6dd59e640 1 -- 172.21.15.39:0/1081295094 >> v1:172.21.15.5:6789/0 conn(0x7fa6d809d250 legacy=0x7fa6d806d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6dd59e640 10 monclient: _finish_hunting 0 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6dd59e640 1 monclient: found mon.noname-c 2024-09-13T11:39:53.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6dd59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:53.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6dd59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.39:3300/0 2024-09-13T11:39:53.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6dd59e640 1 -- 172.21.15.39:0/1081295094 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d80a2ee0 con 0x7fa6d8134a70 2024-09-13T11:39:53.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6dd59e640 1 --2- 172.21.15.39:0/1081295094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6d8134a70 0x7fa6d8134e50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80a2e70 tx=0x7fa6d001c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6637f33c2da6797 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:53.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 1 -- 172.21.15.39:0/1081295094 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d00ab620 con 0x7fa6d8134a70 2024-09-13T11:39:53.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:53.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:39:53.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 10 monclient: dump: 2024-09-13T11:39:53.914 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:53.914 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:53.914 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.914 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.914 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:53.914 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:53.915 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:53.915 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:53.915 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:53.915 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:53.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 10 monclient: _finish_auth 0 2024-09-13T11:39:53.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 10 monclient: _check_auth_tickets 2024-09-13T11:39:53.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:53.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 1 -- 172.21.15.39:0/1081295094 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6d00b0070 con 0x7fa6d8134a70 2024-09-13T11:39:53.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 5 monclient: authenticate success, global_id 4111 2024-09-13T11:39:53.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:39:53.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:53.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 10 monclient: get_monmap_and_config success 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 4 set_mon_vals no callback set 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 1 -- 172.21.15.39:0/1081295094 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d00ab9e0 con 0x7fa6d8134a70 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6d7fff640 10 monclient: dump: 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:53.916 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 10 monclient: shutdown 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 1 -- 172.21.15.39:0/1081295094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6d8134a70 msgr2=0x7fa6d8134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 1 --2- 172.21.15.39:0/1081295094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6d8134a70 0x7fa6d8134e50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80a2e70 tx=0x7fa6d001c400 comp rx=0 tx=0).stop 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 1 -- 172.21.15.39:0/1081295094 shutdown_connections 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 1 --2- 172.21.15.39:0/1081295094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6d8134a70 0x7fa6d8134e50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 1 -- 172.21.15.39:0/1081295094 >> 172.21.15.39:0/1081295094 conn(0x7fa6d80a3e60 msgr2=0x7fa6d80a4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 1 -- 172.21.15.39:0/1081295094 shutdown_connections 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.906+0000 7fa6df802640 1 -- 172.21.15.39:0/1081295094 wait complete. 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient: build_initial_monmap 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient: monmap: 2024-09-13T11:39:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 0 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 0.000000 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:created 0.000000 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 1 Processor -- start 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 1 -- start start 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient: init 2024-09-13T11:39:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6d806da20 0x7fa6d813eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient(hunting): picked mon.noname-a con 0x7fa6d806da20 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6d809d250 0x7fa6d81d0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient(hunting): picked mon.noname-b con 0x7fa6d809d250 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa6d8134a70 0x7fa6d813c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d8134a70 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d8135510 con 0x7fa6d806da20 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d8135390 con 0x7fa6d809d250 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa6d8143bb0 con 0x7fa6d8134a70 2024-09-13T11:39:53.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient(hunting): authenticate will time out at 960.449421s 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dcd9d640 1 -- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6d809d250 msgr2=0x7fa6d81d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dcd9d640 1 -- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6d809d250 msgr2=0x7fa6d81d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dcd9d640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6d809d250 0x7fa6d81d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dcd9d640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6d809d250 0x7fa6d81d0210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6ddd9f640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa6d8134a70 0x7fa6d813c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6d806da20 0x7fa6d813eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6d806da20 0x7fa6d813eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.39:60498/0 (socket says 172.21.15.39:60498) 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 1 -- 172.21.15.39:0/3437136671 learned_addr learned my addr 172.21.15.39:0/3437136671 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 10 monclient(hunting): get_auth_request con 0x7fa6d806da20 auth_method 0 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:53.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6ddd9f640 10 monclient(hunting): get_auth_request con 0x7fa6d8134a70 auth_method 0 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6ddd9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6ddd9f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6ddd9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6ddd9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6ddd9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6ddd9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 10 monclient(hunting): handle_auth_done global_id 4107 payload 824 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 1 -- 172.21.15.39:0/3437136671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa6d8134a70 msgr2=0x7fa6d813c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 1 --2- 172.21.15.39:0/3437136671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa6d8134a70 0x7fa6d813c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 1 -- 172.21.15.39:0/3437136671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6d809d250 msgr2=0x7fa6d81d0210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 1 --2- 172.21.15.39:0/3437136671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6d809d250 0x7fa6d81d0210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 10 monclient: _finish_hunting 0 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 1 monclient: found mon.noname-a 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 1 -- 172.21.15.39:0/3437136671 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d81406c0 con 0x7fa6d806da20 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6dd59e640 1 --2- 172.21.15.39:0/3437136671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6d806da20 0x7fa6d813eec0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa6d001a030 tx=0x7fa6d001ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 1 -- 172.21.15.39:0/3437136671 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d00abce0 con 0x7fa6d806da20 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 10 monclient: dump: 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 10 monclient: _finish_auth 0 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 5 monclient: authenticate success, global_id 4107 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient: _renew_subs 2024-09-13T11:39:53.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 1 -- 172.21.15.39:0/3437136671 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d809e150 con 0x7fa6d806da20 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient: _renew_subs 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 1 -- 172.21.15.39:0/3437136671 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d81403e0 con 0x7fa6d806da20 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 1 -- 172.21.15.39:0/3437136671 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6d00be3f0 con 0x7fa6d806da20 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 1 -- 172.21.15.39:0/3437136671 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d00be7c0 con 0x7fa6d806da20 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d77fe640 4 set_mon_vals no callback set 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 10 monclient: dump: 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:53.924 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 1 -- 172.21.15.39:0/3437136671 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa6d00becf0 con 0x7fa6d806da20 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6d5ffb640 1 -- 172.21.15.39:0/3437136671 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fa6d00b4490 con 0x7fa6d806da20 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.910+0000 7fa6df802640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.914+0000 7fa6df802640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.914+0000 7fa6df802640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:39:53.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.914+0000 7fa6df802640 1 -- 172.21.15.39:0/3437136671 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6b8005090 con 0x7fa6d806da20 2024-09-13T11:39:53.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.918+0000 7fa6d5ffb640 1 -- 172.21.15.39:0/3437136671 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fa6d0016110 con 0x7fa6d806da20 2024-09-13T11:39:53.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.918+0000 7fa6d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:53.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.918+0000 7fa6d5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:39:53.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.990+0000 7fa6df802640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0803d65c-46ec-49b5-a45e-46bcf59ae3f3", "id": 0}] 2024-09-13T11:39:53.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.990+0000 7fa6df802640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0803d65c-46ec-49b5-a45e-46bcf59ae3f3", "id": 0}] 2024-09-13T11:39:53.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.990+0000 7fa6df802640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:39:53.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.990+0000 7fa6df802640 1 -- 172.21.15.39:0/3437136671 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0803d65c-46ec-49b5-a45e-46bcf59ae3f3", "id": 0} v 0) -- 0x7fa6b8005680 con 0x7fa6d806da20 2024-09-13T11:39:53.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6d5ffb640 1 -- 172.21.15.39:0/3437136671 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0803d65c-46ec-49b5-a45e-46bcf59ae3f3", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fa6d00059a0 con 0x7fa6d806da20 2024-09-13T11:39:53.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0803d65c-46ec-49b5-a45e-46bcf59ae3f3", "id": 0}] 2024-09-13T11:39:53.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6d5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:39:53.994 INFO:teuthology.orchestra.run.smithi039.stdout:0 2024-09-13T11:39:53.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6df802640 10 monclient: shutdown 2024-09-13T11:39:53.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6df802640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:53.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6df802640 1 -- 172.21.15.39:0/3437136671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6d806da20 msgr2=0x7fa6d813eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:53.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6df802640 1 --2- 172.21.15.39:0/3437136671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6d806da20 0x7fa6d813eec0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa6d001a030 tx=0x7fa6d001ac40 comp rx=0 tx=0).stop 2024-09-13T11:39:53.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6df802640 1 -- 172.21.15.39:0/3437136671 shutdown_connections 2024-09-13T11:39:53.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6df802640 1 --2- 172.21.15.39:0/3437136671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa6d8134a70 0x7fa6d813c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6df802640 1 --2- 172.21.15.39:0/3437136671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6d809d250 0x7fa6d81d0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6df802640 1 --2- 172.21.15.39:0/3437136671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6d806da20 0x7fa6d813eec0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:53.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6df802640 1 -- 172.21.15.39:0/3437136671 >> 172.21.15.39:0/3437136671 conn(0x7fa6d80a3e60 msgr2=0x7fa6d80a5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:53.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6df802640 1 -- 172.21.15.39:0/3437136671 shutdown_connections 2024-09-13T11:39:53.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:53.994+0000 7fa6df802640 1 -- 172.21.15.39:0/3437136671 wait complete. 2024-09-13T11:39:54.012 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new fd592a78-172c-4e1c-9f3e-c49e3e3304d3 1 2024-09-13T11:39:54.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient: get_monmap_and_config 2024-09-13T11:39:54.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient: build_initial_monmap 2024-09-13T11:39:54.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient: monmap: 2024-09-13T11:39:54.139 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 0 2024-09-13T11:39:54.139 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.139 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:54.140377+0000 2024-09-13T11:39:54.139 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:54.140377+0000 2024-09-13T11:39:54.139 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:54.139 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.139 INFO:teuthology.orchestra.run.smithi039.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:39:54.139 INFO:teuthology.orchestra.run.smithi039.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:39:54.139 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:39:54.139 INFO:teuthology.orchestra.run.smithi039.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:39:54.140 INFO:teuthology.orchestra.run.smithi039.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:39:54.140 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 1 Processor -- start 2024-09-13T11:39:54.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 1 -- start start 2024-09-13T11:39:54.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient: init 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f52c00a3a60 0x7f52c00a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient(hunting): picked mon.noname-a con 0x7f52c00a3a60 addr v2:172.21.15.5:3300/0 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52c00a22b0 addr v1:172.21.15.39:6789/0 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f52c00a4380 0x7f52c0136070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient(hunting): picked mon.noname-c con 0x7f52c00a4380 addr v2:172.21.15.39:3300/0 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52c009fd10 con 0x7f52c00a22b0 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f52c00a1960 con 0x7f52c00a3a60 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f52c009ff90 con 0x7f52c00a4380 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:54.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c7c5f640 10 monclient(hunting): authenticate will time out at 960.679555s 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c61fc640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f52c00a4380 0x7f52c0136070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c61fc640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f52c00a4380 0x7f52c0136070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50642/0 (socket says 172.21.15.39:50642) 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c61fc640 1 -- 172.21.15.39:0/708147084 learned_addr learned my addr 172.21.15.39:0/708147084 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c61fc640 10 monclient(hunting): get_auth_request con 0x7f52c00a4380 auth_method 0 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c61fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c61fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c61fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c61fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c61fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c61fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.138+0000 7f52c59fb640 1 --2- 172.21.15.39:0/708147084 >> v2:172.21.15.5:3300/0 conn(0x7f52c00a3a60 0x7f52c00a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c59fb640 10 monclient(hunting): get_auth_request con 0x7f52c00a3a60 auth_method 0 2024-09-13T11:39:54.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c59fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c59fb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c59fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c61fc640 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c61fc640 1 -- 172.21.15.39:0/708147084 >> v2:172.21.15.5:3300/0 conn(0x7f52c00a3a60 msgr2=0x7f52c00a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c61fc640 1 --2- 172.21.15.39:0/708147084 >> v2:172.21.15.5:3300/0 conn(0x7f52c00a3a60 0x7f52c00a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c61fc640 1 -- 172.21.15.39:0/708147084 >> v1:172.21.15.39:6789/0 conn(0x7f52c00a22b0 legacy=0x7f52c0135960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c61fc640 10 monclient: _finish_hunting 0 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c61fc640 1 monclient: found mon.noname-c 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c61fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c61fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c61fc640 1 -- 172.21.15.39:0/708147084 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c00a1030 con 0x7f52c00a4380 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 1 -- 172.21.15.39:0/708147084 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4279698835 0 0) 0x7f52c009fd10 con 0x7f52c00a22b0 2024-09-13T11:39:54.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:39:54.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c61fc640 1 --2- 172.21.15.39:0/708147084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c00a4380 0x7f52c0136070 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f52c0071310 tx=0x7f52b801d310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=207f6d8219dfd236 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:54.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 1 -- 172.21.15.39:0/708147084 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b80a5790 con 0x7f52c00a4380 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 10 monclient: dump: 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.152 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 10 monclient: _finish_auth 0 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 10 monclient: _check_auth_tickets 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 1 -- 172.21.15.39:0/708147084 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b80a5930 con 0x7f52c00a4380 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 1 -- 172.21.15.39:0/708147084 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b80a5d00 con 0x7f52c00a4380 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 5 monclient: authenticate success, global_id 4117 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 10 monclient: get_monmap_and_config success 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 4 set_mon_vals no callback set 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c49f9640 10 monclient: dump: 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.153 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 10 monclient: shutdown 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 1 -- 172.21.15.39:0/708147084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c00a4380 msgr2=0x7f52c0136070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 1 --2- 172.21.15.39:0/708147084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c00a4380 0x7f52c0136070 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f52c0071310 tx=0x7f52b801d310 comp rx=0 tx=0).stop 2024-09-13T11:39:54.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 1 -- 172.21.15.39:0/708147084 shutdown_connections 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 1 --2- 172.21.15.39:0/708147084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c00a4380 0x7f52c0136070 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 1 --2- 172.21.15.39:0/708147084 >> v2:172.21.15.5:3300/0 conn(0x7f52c00a3a60 0x7f52c00a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 1 -- 172.21.15.39:0/708147084 >> 172.21.15.39:0/708147084 conn(0x7f52c00a8650 msgr2=0x7f52c00a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 1 -- 172.21.15.39:0/708147084 shutdown_connections 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 1 -- 172.21.15.39:0/708147084 wait complete. 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.142+0000 7f52c7c5f640 10 monclient: build_initial_monmap 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient: monmap: 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 0 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 0.000000 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:created 0.000000 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.155 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 1 Processor -- start 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 1 -- start start 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient: init 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52c00a22b0 0x7f52c01cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient(hunting): picked mon.noname-c con 0x7f52c00a22b0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52c00a3a60 0x7f52c01cf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient(hunting): picked mon.noname-a con 0x7f52c00a3a60 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c00a4380 0x7f52c01cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient(hunting): picked mon.noname-b con 0x7f52c00a4380 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f52c01365b0 con 0x7f52c00a3a60 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f52c009f070 con 0x7f52c00a4380 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f52c0136730 con 0x7f52c00a22b0 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient(hunting): authenticate will time out at 960.684988s 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c00a4380 0x7f52c01cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c00a4380 0x7f52c01cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50646/0 (socket says 172.21.15.39:50646) 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 1 -- 172.21.15.39:0/24137802 learned_addr learned my addr 172.21.15.39:0/24137802 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 10 monclient(hunting): get_auth_request con 0x7f52c00a4380 auth_method 0 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c59fb640 1 --2- 172.21.15.39:0/24137802 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52c00a22b0 0x7f52c01cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c51fa640 1 --2- 172.21.15.39:0/24137802 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52c00a3a60 0x7f52c01cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c59fb640 10 monclient(hunting): get_auth_request con 0x7f52c00a22b0 auth_method 0 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c59fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c59fb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c59fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c51fa640 10 monclient(hunting): get_auth_request con 0x7f52c00a3a60 auth_method 0 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c51fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c51fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c51fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 1 -- 172.21.15.39:0/24137802 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52c00a22b0 msgr2=0x7f52c01cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 1 --2- 172.21.15.39:0/24137802 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52c00a22b0 0x7f52c01cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 1 -- 172.21.15.39:0/24137802 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52c00a3a60 msgr2=0x7f52c01cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 1 --2- 172.21.15.39:0/24137802 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52c00a3a60 0x7f52c01cf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 10 monclient: _finish_hunting 0 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 1 monclient: found mon.noname-b 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 1 -- 172.21.15.39:0/24137802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c009e6e0 con 0x7f52c00a4380 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c59fb640 1 --2- 172.21.15.39:0/24137802 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52c00a22b0 0x7f52c01cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:39:54.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c51fa640 1 --2- 172.21.15.39:0/24137802 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52c00a3a60 0x7f52c01cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c61fc640 1 --2- 172.21.15.39:0/24137802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c00a4380 0x7f52c01cfe80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f52b801cd90 tx=0x7f52b8019650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 1 -- 172.21.15.39:0/24137802 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b80bc020 con 0x7f52c00a4380 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 10 monclient: dump: 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 10 monclient: _finish_auth 0 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 10 monclient: _check_auth_tickets 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 5 monclient: authenticate success, global_id 4123 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 1 -- 172.21.15.39:0/24137802 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b80b8670 con 0x7f52c00a4380 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 1 -- 172.21.15.39:0/24137802 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b80b8a20 con 0x7f52c00a4380 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b7fff640 4 set_mon_vals no callback set 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 10 monclient: dump: 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.161 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient: _renew_subs 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 1 -- 172.21.15.39:0/24137802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c0073db0 con 0x7f52c00a4380 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient: _renew_subs 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 1 -- 172.21.15.39:0/24137802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c009d7e0 con 0x7f52c00a4380 2024-09-13T11:39:54.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 1 -- 172.21.15.39:0/24137802 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f52b80bf020 con 0x7f52c00a4380 2024-09-13T11:39:54.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52b67fc640 1 -- 172.21.15.39:0/24137802 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f52b80b67f0 con 0x7f52c00a4380 2024-09-13T11:39:54.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:54.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:54.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.146+0000 7f52c7c5f640 1 -- 172.21.15.39:0/24137802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c00a3e40 con 0x7f52c00a4380 2024-09-13T11:39:54.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.150+0000 7f52b67fc640 1 -- 172.21.15.39:0/24137802 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f52b80c6030 con 0x7f52c00a4380 2024-09-13T11:39:54.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.150+0000 7f52b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:54.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.150+0000 7f52b67fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:39:54.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.222+0000 7f52c7c5f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fd592a78-172c-4e1c-9f3e-c49e3e3304d3", "id": 1}] 2024-09-13T11:39:54.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.222+0000 7f52c7c5f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fd592a78-172c-4e1c-9f3e-c49e3e3304d3", "id": 1}] 2024-09-13T11:39:54.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.222+0000 7f52c7c5f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.222+0000 7f52c7c5f640 1 -- 172.21.15.39:0/24137802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fd592a78-172c-4e1c-9f3e-c49e3e3304d3", "id": 1} v 0) -- 0x7f52c009bfe0 con 0x7f52c00a4380 2024-09-13T11:39:54.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52b67fc640 1 -- 172.21.15.39:0/24137802 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fd592a78-172c-4e1c-9f3e-c49e3e3304d3", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f52b80165c0 con 0x7f52c00a4380 2024-09-13T11:39:54.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fd592a78-172c-4e1c-9f3e-c49e3e3304d3", "id": 1}] 2024-09-13T11:39:54.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52b67fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:39:54.229 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-09-13T11:39:54.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52c7c5f640 10 monclient: shutdown 2024-09-13T11:39:54.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52c7c5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52c7c5f640 1 -- 172.21.15.39:0/24137802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c00a4380 msgr2=0x7f52c01cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52c7c5f640 1 --2- 172.21.15.39:0/24137802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c00a4380 0x7f52c01cfe80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f52b801cd90 tx=0x7f52b8019650 comp rx=0 tx=0).stop 2024-09-13T11:39:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52c7c5f640 1 -- 172.21.15.39:0/24137802 shutdown_connections 2024-09-13T11:39:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52c7c5f640 1 --2- 172.21.15.39:0/24137802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52c00a4380 0x7f52c01cfe80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52c7c5f640 1 --2- 172.21.15.39:0/24137802 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52c00a3a60 0x7f52c01cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52c7c5f640 1 --2- 172.21.15.39:0/24137802 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52c00a22b0 0x7f52c01cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52c7c5f640 1 -- 172.21.15.39:0/24137802 >> 172.21.15.39:0/24137802 conn(0x7f52c00a8650 msgr2=0x7f52c0070480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52c7c5f640 1 -- 172.21.15.39:0/24137802 shutdown_connections 2024-09-13T11:39:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.230+0000 7f52c7c5f640 1 -- 172.21.15.39:0/24137802 wait complete. 2024-09-13T11:39:54.247 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new c3cec868-fd4e-472c-83b8-bc2e466671ee 2 2024-09-13T11:39:54.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient: get_monmap_and_config 2024-09-13T11:39:54.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient: build_initial_monmap 2024-09-13T11:39:54.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient: monmap: 2024-09-13T11:39:54.362 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 0 2024-09-13T11:39:54.362 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.362 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:54.363562+0000 2024-09-13T11:39:54.362 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:54.363562+0000 2024-09-13T11:39:54.362 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:54.362 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.363 INFO:teuthology.orchestra.run.smithi039.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:39:54.363 INFO:teuthology.orchestra.run.smithi039.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:39:54.363 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:39:54.363 INFO:teuthology.orchestra.run.smithi039.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:39:54.363 INFO:teuthology.orchestra.run.smithi039.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:39:54.363 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 1 Processor -- start 2024-09-13T11:39:54.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 1 -- start start 2024-09-13T11:39:54.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient: init 2024-09-13T11:39:54.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:54.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f898409ba40 0x7f898409be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient(hunting): picked mon.noname-b con 0x7f898409ba40 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f898409cea0 0x7f898409c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient(hunting): picked mon.noname-c con 0x7f898409cea0 addr v2:172.21.15.39:3300/0 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8984134790 addr v1:172.21.15.5:6789/0 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f898409d9d0 con 0x7f8984134790 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f898409c8a0 con 0x7f898409cea0 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f898409cc90 con 0x7f898409ba40 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8988c14640 10 monclient(hunting): authenticate will time out at 960.902749s 2024-09-13T11:39:54.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f898409cea0 0x7f898409c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f898409cea0 0x7f898409c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50662/0 (socket says 172.21.15.39:50662) 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 1 -- 172.21.15.39:0/2622246517 learned_addr learned my addr 172.21.15.39:0/2622246517 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 10 monclient(hunting): get_auth_request con 0x7f898409cea0 auth_method 0 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f898259c640 1 --2- 172.21.15.39:0/2622246517 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f898409ba40 0x7f898409be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f898259c640 10 monclient(hunting): get_auth_request con 0x7f898409ba40 auth_method 0 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f898259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f898259c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f898259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2024-09-13T11:39:54.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 1 -- 172.21.15.39:0/2622246517 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f898409ba40 msgr2=0x7f898409be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:39:54.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 1 --2- 172.21.15.39:0/2622246517 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f898409ba40 0x7f898409be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 1 -- 172.21.15.39:0/2622246517 >> v1:172.21.15.5:6789/0 conn(0x7f8984134790 legacy=0x7f8984134c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 10 monclient: _finish_hunting 0 2024-09-13T11:39:54.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 1 monclient: found mon.noname-c 2024-09-13T11:39:54.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:54.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.362+0000 7f8981d9b640 1 -- 172.21.15.39:0/2622246517 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89840a36b0 con 0x7f898409cea0 2024-09-13T11:39:54.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8981d9b640 1 --2- 172.21.15.39:0/2622246517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898409cea0 0x7f898409c360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f896c009870 tx=0x7f896c01c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9c91a09fc7866d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:54.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 1 -- 172.21.15.39:0/2622246517 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c0ab610 con 0x7f898409cea0 2024-09-13T11:39:54.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:39:54.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 10 monclient: dump: 2024-09-13T11:39:54.373 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.373 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.373 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.373 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.373 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.373 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 10 monclient: _finish_auth 0 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 10 monclient: _check_auth_tickets 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 1 -- 172.21.15.39:0/2622246517 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f896c0b0070 con 0x7f898409cea0 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 1 -- 172.21.15.39:0/2622246517 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c0ab9d0 con 0x7f898409cea0 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 5 monclient: authenticate success, global_id 4126 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 10 monclient: get_monmap_and_config success 2024-09-13T11:39:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 4 set_mon_vals no callback set 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f898159a640 10 monclient: dump: 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.375 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 10 monclient: shutdown 2024-09-13T11:39:54.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:54.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 1 -- 172.21.15.39:0/2622246517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898409cea0 msgr2=0x7f898409c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 1 --2- 172.21.15.39:0/2622246517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898409cea0 0x7f898409c360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f896c009870 tx=0x7f896c01c3f0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 1 -- 172.21.15.39:0/2622246517 shutdown_connections 2024-09-13T11:39:54.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 1 --2- 172.21.15.39:0/2622246517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898409cea0 0x7f898409c360 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 1 --2- 172.21.15.39:0/2622246517 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f898409ba40 0x7f898409be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 1 -- 172.21.15.39:0/2622246517 >> 172.21.15.39:0/2622246517 conn(0x7f89840a3ad0 msgr2=0x7f89840a3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:54.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 1 -- 172.21.15.39:0/2622246517 shutdown_connections 2024-09-13T11:39:54.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 1 -- 172.21.15.39:0/2622246517 wait complete. 2024-09-13T11:39:54.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 10 monclient: build_initial_monmap 2024-09-13T11:39:54.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.366+0000 7f8988c14640 10 monclient: monmap: 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 0 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 0.000000 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:created 0.000000 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 1 Processor -- start 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 1 -- start start 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient: init 2024-09-13T11:39:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f898409ba40 0x7f89841334f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient(hunting): picked mon.noname-c con 0x7f898409ba40 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8984134790 0x7f8984133a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient(hunting): picked mon.noname-a con 0x7f8984134790 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898413aba0 0x7f898413afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient(hunting): picked mon.noname-b con 0x7f898413aba0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f898413f400 con 0x7f8984134790 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f898413f280 con 0x7f898413aba0 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f898413f580 con 0x7f898409ba40 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient(hunting): authenticate will time out at 960.908781s 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898413aba0 0x7f898413afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898413aba0 0x7f898413afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50672/0 (socket says 172.21.15.39:50672) 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 1 -- 172.21.15.39:0/2187231096 learned_addr learned my addr 172.21.15.39:0/2187231096 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 10 monclient(hunting): get_auth_request con 0x7f898413aba0 auth_method 0 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:54.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f898259c640 1 --2- 172.21.15.39:0/2187231096 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f898409ba40 0x7f89841334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8981d9b640 1 --2- 172.21.15.39:0/2187231096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8984134790 0x7f8984133a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f898259c640 10 monclient(hunting): get_auth_request con 0x7f898409ba40 auth_method 0 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f898259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f898259c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f898259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8981d9b640 10 monclient(hunting): get_auth_request con 0x7f8984134790 auth_method 0 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8981d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8981d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 1 -- 172.21.15.39:0/2187231096 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f898409ba40 msgr2=0x7f89841334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 1 --2- 172.21.15.39:0/2187231096 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f898409ba40 0x7f89841334f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 1 -- 172.21.15.39:0/2187231096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8984134790 msgr2=0x7f8984133a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 1 --2- 172.21.15.39:0/2187231096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8984134790 0x7f8984133a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 10 monclient: _finish_hunting 0 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 1 monclient: found mon.noname-b 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 1 -- 172.21.15.39:0/2187231096 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898409e1c0 con 0x7f898413aba0 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8982d9d640 1 --2- 172.21.15.39:0/2187231096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898413aba0 0x7f898413afc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f897c00c7c0 tx=0x7f897c00cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 1 -- 172.21.15.39:0/2187231096 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c003b80 con 0x7f898413aba0 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 10 monclient: dump: 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.381 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 10 monclient: _finish_auth 0 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 10 monclient: _check_auth_tickets 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 1 -- 172.21.15.39:0/2187231096 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f897c003d20 con 0x7f898413aba0 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 1 -- 172.21.15.39:0/2187231096 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c00aa90 con 0x7f898413aba0 2024-09-13T11:39:54.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 10 monclient: dump: 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8980d99640 4 set_mon_vals no callback set 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 5 monclient: authenticate success, global_id 4129 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient: _renew_subs 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 1 -- 172.21.15.39:0/2187231096 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8984134450 con 0x7f898413aba0 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient: _renew_subs 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 1 -- 172.21.15.39:0/2187231096 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8984071240 con 0x7f898413aba0 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f8988c14640 1 -- 172.21.15.39:0/2187231096 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8984134c20 con 0x7f898413aba0 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 1 -- 172.21.15.39:0/2187231096 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f897c00ac30 con 0x7f898413aba0 2024-09-13T11:39:54.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.370+0000 7f897affd640 1 -- 172.21.15.39:0/2187231096 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f897c0054f0 con 0x7f898413aba0 2024-09-13T11:39:54.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.374+0000 7f897affd640 1 -- 172.21.15.39:0/2187231096 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f898409d9d0 con 0x7f898413aba0 2024-09-13T11:39:54.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.374+0000 7f897affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:54.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.374+0000 7f897affd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:39:54.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.446+0000 7f8988c14640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c3cec868-fd4e-472c-83b8-bc2e466671ee", "id": 2}] 2024-09-13T11:39:54.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.446+0000 7f8988c14640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c3cec868-fd4e-472c-83b8-bc2e466671ee", "id": 2}] 2024-09-13T11:39:54.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.446+0000 7f8988c14640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.446+0000 7f8988c14640 1 -- 172.21.15.39:0/2187231096 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c3cec868-fd4e-472c-83b8-bc2e466671ee", "id": 2} v 0) -- 0x7f8984070120 con 0x7f898413aba0 2024-09-13T11:39:54.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.450+0000 7f897affd640 1 -- 172.21.15.39:0/2187231096 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c3cec868-fd4e-472c-83b8-bc2e466671ee", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f897c0056f0 con 0x7f898413aba0 2024-09-13T11:39:54.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.450+0000 7f897affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c3cec868-fd4e-472c-83b8-bc2e466671ee", "id": 2}] 2024-09-13T11:39:54.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.450+0000 7f897affd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:39:54.452 INFO:teuthology.orchestra.run.smithi039.stdout:2 2024-09-13T11:39:54.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.454+0000 7f8988c14640 10 monclient: shutdown 2024-09-13T11:39:54.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.454+0000 7f8988c14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:54.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.454+0000 7f8988c14640 1 -- 172.21.15.39:0/2187231096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898413aba0 msgr2=0x7f898413afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.454+0000 7f8988c14640 1 --2- 172.21.15.39:0/2187231096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898413aba0 0x7f898413afc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f897c00c7c0 tx=0x7f897c00cc80 comp rx=0 tx=0).stop 2024-09-13T11:39:54.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.454+0000 7f8988c14640 1 -- 172.21.15.39:0/2187231096 shutdown_connections 2024-09-13T11:39:54.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.454+0000 7f8988c14640 1 --2- 172.21.15.39:0/2187231096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898413aba0 0x7f898413afc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.454+0000 7f8988c14640 1 --2- 172.21.15.39:0/2187231096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8984134790 0x7f8984133a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.454+0000 7f8988c14640 1 --2- 172.21.15.39:0/2187231096 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f898409ba40 0x7f89841334f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.454+0000 7f8988c14640 1 -- 172.21.15.39:0/2187231096 >> 172.21.15.39:0/2187231096 conn(0x7f89840a3ad0 msgr2=0x7f898409fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:54.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.454+0000 7f8988c14640 1 -- 172.21.15.39:0/2187231096 shutdown_connections 2024-09-13T11:39:54.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.454+0000 7f8988c14640 1 -- 172.21.15.39:0/2187231096 wait complete. 2024-09-13T11:39:54.468 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new 748f14f6-a1d6-4669-9f81-6b1cd7cd5a60 3 2024-09-13T11:39:54.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.582+0000 7f1bfb725640 10 monclient: get_monmap_and_config 2024-09-13T11:39:54.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.582+0000 7f1bfb725640 10 monclient: build_initial_monmap 2024-09-13T11:39:54.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.582+0000 7f1bfb725640 10 monclient: monmap: 2024-09-13T11:39:54.584 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 0 2024-09-13T11:39:54.584 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.584 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:54.585677+0000 2024-09-13T11:39:54.584 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:54.585677+0000 2024-09-13T11:39:54.584 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:54.584 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.584 INFO:teuthology.orchestra.run.smithi039.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:39:54.585 INFO:teuthology.orchestra.run.smithi039.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:39:54.585 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:39:54.585 INFO:teuthology.orchestra.run.smithi039.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:39:54.585 INFO:teuthology.orchestra.run.smithi039.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:39:54.585 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 1 Processor -- start 2024-09-13T11:39:54.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 1 -- start start 2024-09-13T11:39:54.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 10 monclient: init 2024-09-13T11:39:54.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:54.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-13T11:39:54.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bf409ba40 addr v1:172.21.15.39:6789/0 2024-09-13T11:39:54.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1bf409cea0 0x7f1bf4139a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf409cea0 addr v2:172.21.15.5:3300/0 2024-09-13T11:39:54.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1bf409c530 0x7f1bf413bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf409c530 addr v2:172.21.15.39:3300/0 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf409d9d0 con 0x7f1bf409ba40 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f1bf409cc90 con 0x7f1bf409cea0 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f1bf40a36b0 con 0x7f1bf409c530 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bfb725640 10 monclient(hunting): authenticate will time out at 961.125084s 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf94c1640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f1bf409ba40 0x7f1bf409be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47418/0 (socket says 172.21.15.39:47418) 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1bf409c530 0x7f1bf413bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf94c1640 1 -- 172.21.15.39:0/4172671616 learned_addr learned my addr 172.21.15.39:0/4172671616 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 10 monclient(hunting): get_auth_request con 0x7f1bf409c530 auth_method 0 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf8cc0640 1 --2- 172.21.15.39:0/4172671616 >> v2:172.21.15.5:3300/0 conn(0x7f1bf409cea0 0x7f1bf4139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf8cc0640 10 monclient(hunting): get_auth_request con 0x7f1bf409cea0 auth_method 0 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bebfff640 1 -- 172.21.15.39:0/4172671616 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3654151918 0 0) 0x7f1bf0001120 con 0x7f1bf409ba40 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf8cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf8cc0640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf8cc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bebfff640 10 monclient(hunting): _negotiate 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bebfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bebfff640 10 monclient(hunting): my global_id is 4135 2024-09-13T11:39:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bebfff640 1 -- 172.21.15.39:0/4172671616 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1be0005740 con 0x7f1bf409ba40 2024-09-13T11:39:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 10 monclient(hunting): handle_auth_done global_id 4132 payload 274 2024-09-13T11:39:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 1 -- 172.21.15.39:0/4172671616 >> v2:172.21.15.5:3300/0 conn(0x7f1bf409cea0 msgr2=0x7f1bf4139a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:39:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 1 --2- 172.21.15.39:0/4172671616 >> v2:172.21.15.5:3300/0 conn(0x7f1bf409cea0 0x7f1bf4139a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 1 -- 172.21.15.39:0/4172671616 >> v1:172.21.15.39:6789/0 conn(0x7f1bf409ba40 legacy=0x7f1bf409be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 10 monclient: _finish_hunting 0 2024-09-13T11:39:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 1 monclient: found mon.noname-c 2024-09-13T11:39:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.586+0000 7f1bf9cc2640 1 -- 172.21.15.39:0/4172671616 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf409dcb0 con 0x7f1bf409c530 2024-09-13T11:39:54.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 1 --2- 172.21.15.39:0/4172671616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bf409c530 0x7f1bf413bf90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40a3180 tx=0x7f1bec01c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79d609fc3efcddb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:54.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 1 -- 172.21.15.39:0/4172671616 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0b2070 con 0x7f1bf409c530 2024-09-13T11:39:54.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:39:54.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 10 monclient: dump: 2024-09-13T11:39:54.596 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.596 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.596 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 10 monclient: _finish_auth 0 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 10 monclient: _check_auth_tickets 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 1 -- 172.21.15.39:0/4172671616 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bec0a57d0 con 0x7f1bf409c530 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 5 monclient: authenticate success, global_id 4132 2024-09-13T11:39:54.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient: get_monmap_and_config success 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 1 -- 172.21.15.39:0/4172671616 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0a5ba0 con 0x7f1bf409c530 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 4 set_mon_vals no callback set 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bebfff640 10 monclient: dump: 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.598 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient: shutdown 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 -- 172.21.15.39:0/4172671616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bf409c530 msgr2=0x7f1bf413bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 --2- 172.21.15.39:0/4172671616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bf409c530 0x7f1bf413bf90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40a3180 tx=0x7f1bec01c7f0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 -- 172.21.15.39:0/4172671616 shutdown_connections 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 --2- 172.21.15.39:0/4172671616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bf409c530 0x7f1bf413bf90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 --2- 172.21.15.39:0/4172671616 >> v2:172.21.15.5:3300/0 conn(0x7f1bf409cea0 0x7f1bf4139a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 -- 172.21.15.39:0/4172671616 >> 172.21.15.39:0/4172671616 conn(0x7f1bf40a3ad0 msgr2=0x7f1bf40a3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 -- 172.21.15.39:0/4172671616 shutdown_connections 2024-09-13T11:39:54.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 -- 172.21.15.39:0/4172671616 wait complete. 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient: build_initial_monmap 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient: monmap: 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 0 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 0.000000 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:created 0.000000 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 Processor -- start 2024-09-13T11:39:54.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 -- start start 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient: init 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bf409ba40 0x7f1bf4131ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf409ba40 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bf409c530 0x7f1bf4131ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf409c530 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bf413ad20 0x7f1bf4136890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf413ad20 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf406deb0 con 0x7f1bf409c530 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf406dd30 con 0x7f1bf413ad20 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf406e030 con 0x7f1bf409ba40 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bfb725640 10 monclient(hunting): authenticate will time out at 961.131219s 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bf413ad20 0x7f1bf4136890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bf413ad20 0x7f1bf4136890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55000/0 (socket says 172.21.15.39:55000) 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 1 -- 172.21.15.39:0/1143562346 learned_addr learned my addr 172.21.15.39:0/1143562346 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 10 monclient(hunting): get_auth_request con 0x7f1bf413ad20 auth_method 0 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:54.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf8cc0640 1 --2- 172.21.15.39:0/1143562346 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bf409c530 0x7f1bf4131ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf94c1640 1 --2- 172.21.15.39:0/1143562346 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bf409ba40 0x7f1bf4131ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 1 -- 172.21.15.39:0/1143562346 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bf409ba40 msgr2=0x7f1bf4131ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 1 --2- 172.21.15.39:0/1143562346 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bf409ba40 0x7f1bf4131ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 1 -- 172.21.15.39:0/1143562346 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bf409c530 msgr2=0x7f1bf4131ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 1 --2- 172.21.15.39:0/1143562346 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bf409c530 0x7f1bf4131ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 10 monclient: _finish_hunting 0 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 1 monclient: found mon.noname-b 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 1 -- 172.21.15.39:0/1143562346 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf40a50f0 con 0x7f1bf413ad20 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.590+0000 7f1bf9cc2640 1 --2- 172.21.15.39:0/1143562346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bf413ad20 0x7f1bf4136890 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1bec019400 tx=0x7f1bec019590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:54.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 1 -- 172.21.15.39:0/1143562346 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0b2050 con 0x7f1bf413ad20 2024-09-13T11:39:54.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:39:54.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 10 monclient: dump: 2024-09-13T11:39:54.604 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.604 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.604 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.604 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.604 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.604 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.604 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.604 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.604 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.605 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 10 monclient: _finish_auth 0 2024-09-13T11:39:54.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:39:54.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:54.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 1 -- 172.21.15.39:0/1143562346 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bec0be070 con 0x7f1bf413ad20 2024-09-13T11:39:54.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:54.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 1 -- 172.21.15.39:0/1143562346 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec009450 con 0x7f1bf413ad20 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 10 monclient: dump: 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1bfb725640 5 monclient: authenticate success, global_id 4138 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1beb7fe640 4 set_mon_vals no callback set 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1bfb725640 10 monclient: _renew_subs 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1bfb725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1bfb725640 1 -- 172.21.15.39:0/1143562346 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf413c190 con 0x7f1bf413ad20 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1bfb725640 10 monclient: _renew_subs 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1bfb725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1bfb725640 1 -- 172.21.15.39:0/1143562346 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf4139fe0 con 0x7f1bf413ad20 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 1 -- 172.21.15.39:0/1143562346 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1bec0ad890 con 0x7f1bf413ad20 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 1 -- 172.21.15.39:0/1143562346 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f1bec0b9450 con 0x7f1bf413ad20 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1bfb725640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1bfb725640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1bfb725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1bfb725640 1 -- 172.21.15.39:0/1143562346 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf409be20 con 0x7f1bf413ad20 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 1 -- 172.21.15.39:0/1143562346 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f1bf40a50f0 con 0x7f1bf413ad20 2024-09-13T11:39:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:54.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.594+0000 7f1be9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:39:54.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.674+0000 7f1bfb725640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "748f14f6-a1d6-4669-9f81-6b1cd7cd5a60", "id": 3}] 2024-09-13T11:39:54.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.674+0000 7f1bfb725640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "748f14f6-a1d6-4669-9f81-6b1cd7cd5a60", "id": 3}] 2024-09-13T11:39:54.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.674+0000 7f1bfb725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.674+0000 7f1bfb725640 1 -- 172.21.15.39:0/1143562346 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd new", "uuid": "748f14f6-a1d6-4669-9f81-6b1cd7cd5a60", "id": 3} v 0) -- 0x7f1bf4132530 con 0x7f1bf413ad20 2024-09-13T11:39:54.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.678+0000 7f1be9ffb640 1 -- 172.21.15.39:0/1143562346 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "748f14f6-a1d6-4669-9f81-6b1cd7cd5a60", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f1bec016500 con 0x7f1bf413ad20 2024-09-13T11:39:54.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.678+0000 7f1be9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "748f14f6-a1d6-4669-9f81-6b1cd7cd5a60", "id": 3}] 2024-09-13T11:39:54.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.678+0000 7f1be9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:39:54.680 INFO:teuthology.orchestra.run.smithi039.stdout:3 2024-09-13T11:39:54.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.678+0000 7f1bfb725640 10 monclient: shutdown 2024-09-13T11:39:54.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.682+0000 7f1bfb725640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:54.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.682+0000 7f1bfb725640 1 -- 172.21.15.39:0/1143562346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bf413ad20 msgr2=0x7f1bf4136890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.682+0000 7f1bfb725640 1 --2- 172.21.15.39:0/1143562346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bf413ad20 0x7f1bf4136890 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1bec019400 tx=0x7f1bec019590 comp rx=0 tx=0).stop 2024-09-13T11:39:54.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.682+0000 7f1bfb725640 1 -- 172.21.15.39:0/1143562346 shutdown_connections 2024-09-13T11:39:54.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.682+0000 7f1bfb725640 1 --2- 172.21.15.39:0/1143562346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bf413ad20 0x7f1bf4136890 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.682+0000 7f1bfb725640 1 --2- 172.21.15.39:0/1143562346 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bf409c530 0x7f1bf4131ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.682+0000 7f1bfb725640 1 --2- 172.21.15.39:0/1143562346 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1bf409ba40 0x7f1bf4131ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.682+0000 7f1bfb725640 1 -- 172.21.15.39:0/1143562346 >> 172.21.15.39:0/1143562346 conn(0x7f1bf40a3ad0 msgr2=0x7f1bf40a5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:54.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.682+0000 7f1bfb725640 1 -- 172.21.15.39:0/1143562346 shutdown_connections 2024-09-13T11:39:54.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.682+0000 7f1bfb725640 1 -- 172.21.15.39:0/1143562346 wait complete. 2024-09-13T11:39:54.696 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new f3a05376-7e83-46f9-b3e6-97b31a0942a8 4 2024-09-13T11:39:54.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.802+0000 7f5ace304640 10 monclient: get_monmap_and_config 2024-09-13T11:39:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.802+0000 7f5ace304640 10 monclient: build_initial_monmap 2024-09-13T11:39:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.802+0000 7f5ace304640 10 monclient: monmap: 2024-09-13T11:39:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 0 2024-09-13T11:39:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:54.805451+0000 2024-09-13T11:39:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:54.805451+0000 2024-09-13T11:39:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:39:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:39:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:39:54.805 INFO:teuthology.orchestra.run.smithi039.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:39:54.805 INFO:teuthology.orchestra.run.smithi039.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:39:54.805 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 1 Processor -- start 2024-09-13T11:39:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 1 -- start start 2024-09-13T11:39:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 10 monclient: init 2024-09-13T11:39:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-13T11:39:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ac80739d0 addr v1:172.21.15.5:6789/0 2024-09-13T11:39:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5ac8134570 0x7f5ac813c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac8134570 addr v2:172.21.15.5:3300/0 2024-09-13T11:39:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5ac813ceb0 0x7f5ac813f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac813ceb0 addr v2:172.21.15.39:3300/0 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ac81349c0 con 0x7f5ac80739d0 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac806f9d0 con 0x7f5ac8134570 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac806fc90 con 0x7f5ac813ceb0 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 10 monclient(hunting): authenticate will time out at 961.345198s 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5ac813ceb0 0x7f5ac813f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5ac813ceb0 0x7f5ac813f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55012/0 (socket says 172.21.15.39:55012) 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 1 -- 172.21.15.39:0/2355697190 learned_addr learned my addr 172.21.15.39:0/2355697190 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 10 monclient(hunting): get_auth_request con 0x7f5ac813ceb0 auth_method 0 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac77fe640 1 --2- 172.21.15.39:0/2355697190 >> v2:172.21.15.5:3300/0 conn(0x7f5ac8134570 0x7f5ac813c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac77fe640 10 monclient(hunting): get_auth_request con 0x7f5ac8134570 auth_method 0 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 1 -- 172.21.15.39:0/2355697190 >> v2:172.21.15.5:3300/0 conn(0x7f5ac8134570 msgr2=0x7f5ac813c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 1 --2- 172.21.15.39:0/2355697190 >> v2:172.21.15.5:3300/0 conn(0x7f5ac8134570 0x7f5ac813c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 1 -- 172.21.15.39:0/2355697190 >> v1:172.21.15.5:6789/0 conn(0x7f5ac80739d0 legacy=0x7f5ac8073db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:39:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 10 monclient: _finish_hunting 0 2024-09-13T11:39:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 1 monclient: found mon.noname-c 2024-09-13T11:39:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 1 -- 172.21.15.39:0/2355697190 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac806ff70 con 0x7f5ac813ceb0 2024-09-13T11:39:54.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5acc8a1640 1 --2- 172.21.15.39:0/2355697190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac813ceb0 0x7f5ac813f2a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5ac809ffe0 tx=0x7f5ac001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cf792702f8055cc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:54.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 1 -- 172.21.15.39:0/2355697190 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00a5730 con 0x7f5ac813ceb0 2024-09-13T11:39:54.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:39:54.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 10 monclient: dump: 2024-09-13T11:39:54.815 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.815 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.815 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.815 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.815 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.815 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.815 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.815 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.815 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 10 monclient: _finish_auth 0 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 1 -- 172.21.15.39:0/2355697190 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac00a58d0 con 0x7f5ac813ceb0 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 5 monclient: authenticate success, global_id 4141 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 10 monclient: get_monmap_and_config success 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 1 -- 172.21.15.39:0/2355697190 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00a5ca0 con 0x7f5ac813ceb0 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ace304640 4 set_mon_vals no callback set 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.806+0000 7f5ac6ffd640 10 monclient: dump: 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient: shutdown 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:54.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 -- 172.21.15.39:0/2355697190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac813ceb0 msgr2=0x7f5ac813f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 --2- 172.21.15.39:0/2355697190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac813ceb0 0x7f5ac813f2a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5ac809ffe0 tx=0x7f5ac001c890 comp rx=0 tx=0).stop 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 -- 172.21.15.39:0/2355697190 shutdown_connections 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 --2- 172.21.15.39:0/2355697190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac813ceb0 0x7f5ac813f2a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 --2- 172.21.15.39:0/2355697190 >> v2:172.21.15.5:3300/0 conn(0x7f5ac8134570 0x7f5ac813c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 -- 172.21.15.39:0/2355697190 >> 172.21.15.39:0/2355697190 conn(0x7f5ac80a6e60 msgr2=0x7f5ac80a3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 -- 172.21.15.39:0/2355697190 shutdown_connections 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 -- 172.21.15.39:0/2355697190 wait complete. 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient: build_initial_monmap 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient: monmap: 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 0 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 0.000000 2024-09-13T11:39:54.818 INFO:teuthology.orchestra.run.smithi039.stderr:created 0.000000 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 Processor -- start 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 -- start start 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient: init 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac80739d0 0x7f5ac81cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient(hunting): picked mon.noname-b con 0x7f5ac80739d0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ac8134570 0x7f5ac81331a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac8134570 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ac813ceb0 0x7f5ac81395d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac813ceb0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac8142180 con 0x7f5ac813ceb0 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac8142000 con 0x7f5ac80739d0 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac8142300 con 0x7f5ac8134570 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient(hunting): authenticate will time out at 961.350137s 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac80739d0 0x7f5ac81cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac80739d0 0x7f5ac81cf5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55014/0 (socket says 172.21.15.39:55014) 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 1 -- 172.21.15.39:0/2526773377 learned_addr learned my addr 172.21.15.39:0/2526773377 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 10 monclient(hunting): get_auth_request con 0x7f5ac80739d0 auth_method 0 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5acc8a1640 1 --2- 172.21.15.39:0/2526773377 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ac813ceb0 0x7f5ac81395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac77fe640 1 --2- 172.21.15.39:0/2526773377 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ac8134570 0x7f5ac81331a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac77fe640 10 monclient(hunting): get_auth_request con 0x7f5ac8134570 auth_method 0 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5acc8a1640 10 monclient(hunting): get_auth_request con 0x7f5ac813ceb0 auth_method 0 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5acc8a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5acc8a1640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5acc8a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 1 -- 172.21.15.39:0/2526773377 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ac8134570 msgr2=0x7f5ac81331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 1 --2- 172.21.15.39:0/2526773377 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ac8134570 0x7f5ac81331a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 1 -- 172.21.15.39:0/2526773377 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ac813ceb0 msgr2=0x7f5ac81395d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 1 --2- 172.21.15.39:0/2526773377 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ac813ceb0 0x7f5ac81395d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 10 monclient: _finish_hunting 0 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 1 monclient: found mon.noname-b 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:54.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 1 -- 172.21.15.39:0/2526773377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac813c8f0 con 0x7f5ac80739d0 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac7fff640 1 --2- 172.21.15.39:0/2526773377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac80739d0 0x7f5ac81cf5d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800c450 tx=0x7f5ab800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 1 -- 172.21.15.39:0/2526773377 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8012830 con 0x7f5ac80739d0 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 10 monclient: dump: 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.823 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 10 monclient: _finish_auth 0 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 1 -- 172.21.15.39:0/2526773377 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab8017070 con 0x7f5ac80739d0 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 1 -- 172.21.15.39:0/2526773377 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8012c30 con 0x7f5ac80739d0 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 10 monclient: dump: 2024-09-13T11:39:54.824 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 5 monclient: authenticate success, global_id 4144 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient: _renew_subs 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 -- 172.21.15.39:0/2526773377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac8074a00 con 0x7f5ac80739d0 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac67fc640 4 set_mon_vals no callback set 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient: _renew_subs 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 -- 172.21.15.39:0/2526773377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac811d290 con 0x7f5ac80739d0 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 1 -- 172.21.15.39:0/2526773377 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5ab801c460 con 0x7f5ac80739d0 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ac4ff9640 1 -- 172.21.15.39:0/2526773377 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f5ab8012dd0 con 0x7f5ac80739d0 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.810+0000 7f5ace304640 1 -- 172.21.15.39:0/2526773377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac8073db0 con 0x7f5ac80739d0 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.814+0000 7f5ac4ff9640 1 -- 172.21.15.39:0/2526773377 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f5ab801b690 con 0x7f5ac80739d0 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.814+0000 7f5ac4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:54.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.814+0000 7f5ac4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:39:54.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.886+0000 7f5ace304640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f3a05376-7e83-46f9-b3e6-97b31a0942a8", "id": 4}] 2024-09-13T11:39:54.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.886+0000 7f5ace304640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f3a05376-7e83-46f9-b3e6-97b31a0942a8", "id": 4}] 2024-09-13T11:39:54.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.886+0000 7f5ace304640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:54.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.886+0000 7f5ace304640 1 -- 172.21.15.39:0/2526773377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f3a05376-7e83-46f9-b3e6-97b31a0942a8", "id": 4} v 0) -- 0x7f5ac811ce60 con 0x7f5ac80739d0 2024-09-13T11:39:54.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.890+0000 7f5ac4ff9640 1 -- 172.21.15.39:0/2526773377 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f3a05376-7e83-46f9-b3e6-97b31a0942a8", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f5ab801b9a0 con 0x7f5ac80739d0 2024-09-13T11:39:54.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.890+0000 7f5ac4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f3a05376-7e83-46f9-b3e6-97b31a0942a8", "id": 4}] 2024-09-13T11:39:54.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.890+0000 7f5ac4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:39:54.893 INFO:teuthology.orchestra.run.smithi039.stdout:4 2024-09-13T11:39:54.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.894+0000 7f5ace304640 10 monclient: shutdown 2024-09-13T11:39:54.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.894+0000 7f5ace304640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:54.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.894+0000 7f5ace304640 1 -- 172.21.15.39:0/2526773377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac80739d0 msgr2=0x7f5ac81cf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:54.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.894+0000 7f5ace304640 1 --2- 172.21.15.39:0/2526773377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac80739d0 0x7f5ac81cf5d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800c450 tx=0x7f5ab800c910 comp rx=0 tx=0).stop 2024-09-13T11:39:54.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.894+0000 7f5ace304640 1 -- 172.21.15.39:0/2526773377 shutdown_connections 2024-09-13T11:39:54.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.894+0000 7f5ace304640 1 --2- 172.21.15.39:0/2526773377 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5ac813ceb0 0x7f5ac81395d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.894+0000 7f5ace304640 1 --2- 172.21.15.39:0/2526773377 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5ac8134570 0x7f5ac81331a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.894+0000 7f5ace304640 1 --2- 172.21.15.39:0/2526773377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac80739d0 0x7f5ac81cf5d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:54.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.894+0000 7f5ace304640 1 -- 172.21.15.39:0/2526773377 >> 172.21.15.39:0/2526773377 conn(0x7f5ac80a6e60 msgr2=0x7f5ac80a2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:54.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.894+0000 7f5ace304640 1 -- 172.21.15.39:0/2526773377 shutdown_connections 2024-09-13T11:39:54.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:54.894+0000 7f5ace304640 1 -- 172.21.15.39:0/2526773377 wait complete. 2024-09-13T11:39:54.909 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new e3b64b2d-c3e5-45d1-90a9-225892061800 5 2024-09-13T11:39:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.018+0000 7f0ca6583640 10 monclient: get_monmap_and_config 2024-09-13T11:39:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.018+0000 7f0ca6583640 10 monclient: build_initial_monmap 2024-09-13T11:39:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.018+0000 7f0ca6583640 10 monclient: monmap: 2024-09-13T11:39:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 0 2024-09-13T11:39:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:55.020847+0000 2024-09-13T11:39:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:55.020847+0000 2024-09-13T11:39:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:55.020 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:55.020 INFO:teuthology.orchestra.run.smithi039.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:39:55.020 INFO:teuthology.orchestra.run.smithi039.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:39:55.020 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:39:55.020 INFO:teuthology.orchestra.run.smithi039.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:39:55.020 INFO:teuthology.orchestra.run.smithi039.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:39:55.020 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:55.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.018+0000 7f0ca6583640 1 Processor -- start 2024-09-13T11:39:55.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.018+0000 7f0ca6583640 1 -- start start 2024-09-13T11:39:55.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.018+0000 7f0ca6583640 10 monclient: init 2024-09-13T11:39:55.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0ca006d310 0x7f0ca009fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 10 monclient(hunting): picked mon.noname-c con 0x7f0ca006d310 addr v2:172.21.15.39:3300/0 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ca006d6f0 addr v1:172.21.15.39:6789/0 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ca013ab80 0x7f0ca013cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 10 monclient(hunting): picked mon.noname-b con 0x7f0ca013ab80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ca00a3060 con 0x7f0ca006d6f0 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f0ca009f3e0 con 0x7f0ca006d310 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca009f100 con 0x7f0ca013ab80 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 10 monclient(hunting): authenticate will time out at 961.560219s 2024-09-13T11:39:55.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0ca006d310 0x7f0ca009fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0ca006d310 0x7f0ca009fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55024/0 (socket says 172.21.15.39:55024) 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9f7fe640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f0ca006d6f0 0x7f0ca006db00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47420/0 (socket says 172.21.15.39:47420) 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 1 -- 172.21.15.39:0/1077725643 learned_addr learned my addr 172.21.15.39:0/1077725643 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 10 monclient(hunting): get_auth_request con 0x7f0ca006d310 auth_method 0 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca4b20640 1 --2- 172.21.15.39:0/1077725643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ca013ab80 0x7f0ca013cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca4b20640 10 monclient(hunting): get_auth_request con 0x7f0ca013ab80 auth_method 0 2024-09-13T11:39:55.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca4b20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:55.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca4b20640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:55.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca4b20640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:55.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 10 monclient(hunting): handle_auth_done global_id 4147 payload 274 2024-09-13T11:39:55.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 1 -- 172.21.15.39:0/1077725643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ca013ab80 msgr2=0x7f0ca013cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:39:55.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 1 --2- 172.21.15.39:0/1077725643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ca013ab80 0x7f0ca013cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 1 -- 172.21.15.39:0/1077725643 >> v1:172.21.15.39:6789/0 conn(0x7f0ca006d6f0 legacy=0x7f0ca006db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:55.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 10 monclient: _finish_hunting 0 2024-09-13T11:39:55.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 1 monclient: found mon.noname-c 2024-09-13T11:39:55.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:55.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.39:3300/0 2024-09-13T11:39:55.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 1 -- 172.21.15.39:0/1077725643 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca009ee20 con 0x7f0ca006d310 2024-09-13T11:39:55.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9ffff640 1 --2- 172.21.15.39:0/1077725643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ca006d310 0x7f0ca009fae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0c940098a0 tx=0x7f0c9401c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8cbb7abc9e33887 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 1 -- 172.21.15.39:0/1077725643 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940b0070 con 0x7f0ca006d310 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 10 monclient: dump: 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:55.030 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 10 monclient: _finish_auth 0 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 10 monclient: _check_auth_tickets 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 1 -- 172.21.15.39:0/1077725643 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c940ab600 con 0x7f0ca006d310 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 5 monclient: authenticate success, global_id 4147 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 10 monclient: get_monmap_and_config success 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 1 -- 172.21.15.39:0/1077725643 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940ab9d0 con 0x7f0ca006d310 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 4 set_mon_vals no callback set 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0c9effd640 10 monclient: dump: 2024-09-13T11:39:55.031 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 10 monclient: shutdown 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 1 -- 172.21.15.39:0/1077725643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ca006d310 msgr2=0x7f0ca009fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:55.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 1 --2- 172.21.15.39:0/1077725643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ca006d310 0x7f0ca009fae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0c940098a0 tx=0x7f0c9401c400 comp rx=0 tx=0).stop 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 1 -- 172.21.15.39:0/1077725643 shutdown_connections 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 1 --2- 172.21.15.39:0/1077725643 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ca013ab80 0x7f0ca013cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 1 --2- 172.21.15.39:0/1077725643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ca006d310 0x7f0ca009fae0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.022+0000 7f0ca6583640 1 -- 172.21.15.39:0/1077725643 >> 172.21.15.39:0/1077725643 conn(0x7f0ca00a3c30 msgr2=0x7f0ca00a4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 -- 172.21.15.39:0/1077725643 shutdown_connections 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 -- 172.21.15.39:0/1077725643 wait complete. 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient: build_initial_monmap 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient: monmap: 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 0 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 0.000000 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:created 0.000000 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:55.033 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 Processor -- start 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 -- start start 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient: init 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ca006d310 0x7f0ca011d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient(hunting): picked mon.noname-b con 0x7f0ca006d310 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ca006d6f0 0x7f0ca011daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:55.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient(hunting): picked mon.noname-c con 0x7f0ca006d6f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ca013ab80 0x7f0ca011e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient(hunting): picked mon.noname-a con 0x7f0ca013ab80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca01400b0 con 0x7f0ca013ab80 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca013ff30 con 0x7f0ca006d310 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca0140230 con 0x7f0ca006d6f0 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient(hunting): authenticate will time out at 961.565444s 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ca006d310 0x7f0ca011d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ca006d310 0x7f0ca011d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55030/0 (socket says 172.21.15.39:55030) 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 1 -- 172.21.15.39:0/822023511 learned_addr learned my addr 172.21.15.39:0/822023511 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-09-13T11:39:55.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 10 monclient(hunting): get_auth_request con 0x7f0ca006d310 auth_method 0 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 10 monclient(hunting): handle_auth_done global_id 4153 payload 824 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca4b20640 1 --2- 172.21.15.39:0/822023511 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ca013ab80 0x7f0ca011e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 1 -- 172.21.15.39:0/822023511 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ca006d6f0 msgr2=0x7f0ca011daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 1 --2- 172.21.15.39:0/822023511 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ca006d6f0 0x7f0ca011daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 1 -- 172.21.15.39:0/822023511 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ca013ab80 msgr2=0x7f0ca011e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 1 --2- 172.21.15.39:0/822023511 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ca013ab80 0x7f0ca011e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 10 monclient: _finish_hunting 0 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 1 monclient: found mon.noname-b 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.39:3300/0 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 1 -- 172.21.15.39:0/822023511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca013d0c0 con 0x7f0ca006d310 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca4b20640 10 monclient: get_auth_request con 0x7f0ca013ab80 auth_method 0 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca4b20640 1 --2- 172.21.15.39:0/822023511 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ca013ab80 0x7f0ca011e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9ffff640 1 --2- 172.21.15.39:0/822023511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ca006d310 0x7f0ca011d5b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0c9401a030 tx=0x7f0c940191a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 1 -- 172.21.15.39:0/822023511 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940b0040 con 0x7f0ca006d310 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 10 monclient: dump: 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:55.037 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 10 monclient: _finish_auth 0 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 10 monclient: _check_auth_tickets 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 1 -- 172.21.15.39:0/822023511 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c940ab400 con 0x7f0ca006d310 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 1 -- 172.21.15.39:0/822023511 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940b4660 con 0x7f0ca006d310 2024-09-13T11:39:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 10 monclient: dump: 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:epoch 1 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:election_strategy: 1 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9e7fc640 4 set_mon_vals no callback set 2024-09-13T11:39:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 5 monclient: authenticate success, global_id 4153 2024-09-13T11:39:55.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient: _renew_subs 2024-09-13T11:39:55.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:55.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 -- 172.21.15.39:0/822023511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ca0073db0 con 0x7f0ca006d310 2024-09-13T11:39:55.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient: _renew_subs 2024-09-13T11:39:55.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:55.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 -- 172.21.15.39:0/822023511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ca011fd00 con 0x7f0ca006d310 2024-09-13T11:39:55.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 1 -- 172.21.15.39:0/822023511 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0c940bb020 con 0x7f0ca006d310 2024-09-13T11:39:55.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0c9cff9640 1 -- 172.21.15.39:0/822023511 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f0c940bdd50 con 0x7f0ca006d310 2024-09-13T11:39:55.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:55.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:55.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:55.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.026+0000 7f0ca6583640 1 -- 172.21.15.39:0/822023511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ca009fae0 con 0x7f0ca006d310 2024-09-13T11:39:55.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.030+0000 7f0c9cff9640 1 -- 172.21.15.39:0/822023511 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f0c940c2020 con 0x7f0ca006d310 2024-09-13T11:39:55.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.030+0000 7f0c9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:55.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.030+0000 7f0c9cff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:39:55.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.102+0000 7f0ca6583640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e3b64b2d-c3e5-45d1-90a9-225892061800", "id": 5}] 2024-09-13T11:39:55.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.102+0000 7f0ca6583640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e3b64b2d-c3e5-45d1-90a9-225892061800", "id": 5}] 2024-09-13T11:39:55.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.102+0000 7f0ca6583640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:39:55.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.102+0000 7f0ca6583640 1 -- 172.21.15.39:0/822023511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e3b64b2d-c3e5-45d1-90a9-225892061800", "id": 5} v 0) -- 0x7f0ca006cbe0 con 0x7f0ca006d310 2024-09-13T11:39:55.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0c9cff9640 1 -- 172.21.15.39:0/822023511 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e3b64b2d-c3e5-45d1-90a9-225892061800", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f0c94016110 con 0x7f0ca006d310 2024-09-13T11:39:55.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0c9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e3b64b2d-c3e5-45d1-90a9-225892061800", "id": 5}] 2024-09-13T11:39:55.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0c9cff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:39:55.109 INFO:teuthology.orchestra.run.smithi039.stdout:5 2024-09-13T11:39:55.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0ca6583640 10 monclient: shutdown 2024-09-13T11:39:55.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0ca6583640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:55.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0ca6583640 1 -- 172.21.15.39:0/822023511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ca006d310 msgr2=0x7f0ca011d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:55.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0ca6583640 1 --2- 172.21.15.39:0/822023511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ca006d310 0x7f0ca011d5b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0c9401a030 tx=0x7f0c940191a0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0ca6583640 1 -- 172.21.15.39:0/822023511 shutdown_connections 2024-09-13T11:39:55.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0ca6583640 1 --2- 172.21.15.39:0/822023511 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ca013ab80 0x7f0ca011e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0ca6583640 1 --2- 172.21.15.39:0/822023511 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ca006d6f0 0x7f0ca011daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0ca6583640 1 --2- 172.21.15.39:0/822023511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ca006d310 0x7f0ca011d5b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0ca6583640 1 -- 172.21.15.39:0/822023511 >> 172.21.15.39:0/822023511 conn(0x7f0ca00a3c30 msgr2=0x7f0ca013c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:55.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0ca6583640 1 -- 172.21.15.39:0/822023511 shutdown_connections 2024-09-13T11:39:55.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T11:39:55.110+0000 7f0ca6583640 1 -- 172.21.15.39:0/822023511 wait complete. 2024-09-13T11:39:55.126 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T11:39:55.126 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T11:39:55.128 INFO:tasks.ceph.osd.0:Started 2024-09-13T11:39:55.128 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T11:39:55.129 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T11:39:55.131 INFO:tasks.ceph.osd.1:Started 2024-09-13T11:39:55.131 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T11:39:55.131 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T11:39:55.133 INFO:tasks.ceph.osd.2:Started 2024-09-13T11:39:55.134 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T11:39:55.134 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T11:39:55.135 INFO:tasks.ceph.osd.3:Started 2024-09-13T11:39:55.136 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T11:39:55.136 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T11:39:55.137 INFO:tasks.ceph.osd.4:Started 2024-09-13T11:39:55.138 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-13T11:39:55.138 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-13T11:39:55.140 INFO:tasks.ceph.osd.5:Started 2024-09-13T11:39:55.140 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:39:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.263+0000 7f26315f5640 10 monclient: get_monmap_and_config 2024-09-13T11:39:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.263+0000 7f26315f5640 10 monclient: build_initial_monmap 2024-09-13T11:39:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 10 monclient: monmap: 2024-09-13T11:39:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:39:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:55.268842+0000 2024-09-13T11:39:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:55.268842+0000 2024-09-13T11:39:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:39:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:39:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:39:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:39:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:39:55.268 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 1 Processor -- start 2024-09-13T11:39:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 1 -- start start 2024-09-13T11:39:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 10 monclient: init 2024-09-13T11:39:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-13T11:39:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 0x7f262c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 10 monclient(hunting): picked mon.noname-b con 0x7f262c137930 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f262c137d10 addr v1:172.21.15.5:6789/0 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f262c13cd80 0x7f262c13f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 10 monclient(hunting): picked mon.noname-a con 0x7f262c13cd80 addr v2:172.21.15.5:3300/0 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f262c0a5a80 con 0x7f262c137d10 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f262c0a5fe0 con 0x7f262c13cd80 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f262c09c540 con 0x7f262c137930 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f26315f5640 10 monclient(hunting): authenticate will time out at 998.617186s 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f262c13cd80 0x7f262c13f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262affd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 0x7f262c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f262c13cd80 0x7f262c13f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47052/0 (socket says 172.21.15.5:47052) 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262affd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 0x7f262c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:43004/0 (socket says 172.21.15.5:43004) 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 1 -- 172.21.15.5:0/62795345 learned_addr learned my addr 172.21.15.5:0/62795345 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262a7fc640 1 --1- 172.21.15.5:0/62795345 >> v1:172.21.15.5:6789/0 conn(0x7f262c137d10 0x7f262c1380f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:53404/0 (socket says 172.21.15.5:53404) 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 10 monclient(hunting): get_auth_request con 0x7f262c13cd80 auth_method 0 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262affd640 10 monclient(hunting): get_auth_request con 0x7f262c137930 auth_method 0 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 10 monclient(hunting): handle_auth_done global_id 4152 payload 274 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 1 -- 172.21.15.5:0/62795345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 msgr2=0x7f262c09cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 1 --2- 172.21.15.5:0/62795345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 0x7f262c09cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 1 -- 172.21.15.5:0/62795345 >> v1:172.21.15.5:6789/0 conn(0x7f262c137d10 legacy=0x7f262c1380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 1 monclient: found mon.noname-a 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:39:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f2629ffb640 1 -- 172.21.15.5:0/62795345 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2585789771 0 0) 0x7f262c0a5a80 con 0x7f262c137d10 2024-09-13T11:39:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f262b7fe640 1 -- 172.21.15.5:0/62795345 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f262c09bef0 con 0x7f262c13cd80 2024-09-13T11:39:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.267+0000 7f2629ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:39:55.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262b7fe640 1 --2- 172.21.15.5:0/62795345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f262c13cd80 0x7f262c13f170 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2624009e50 tx=0x7f262401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb592d614a03d6a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 1 -- 172.21.15.5:0/62795345 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26240a57a0 con 0x7f262c13cd80 2024-09-13T11:39:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:39:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 10 monclient: dump: 2024-09-13T11:39:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:39:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:55.280 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 10 monclient: _finish_auth 0 2024-09-13T11:39:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:39:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 1 -- 172.21.15.5:0/62795345 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26240a5940 con 0x7f262c13cd80 2024-09-13T11:39:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 1 -- 172.21.15.5:0/62795345 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26240a5d10 con 0x7f262c13cd80 2024-09-13T11:39:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 5 monclient: authenticate success, global_id 4152 2024-09-13T11:39:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient: get_monmap_and_config success 2024-09-13T11:39:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 4 set_mon_vals no callback set 2024-09-13T11:39:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:39:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f2629ffb640 10 monclient: dump: 2024-09-13T11:39:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:39:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:55.282 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:55.282 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:55.282 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:55.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient: shutdown 2024-09-13T11:39:55.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:55.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 -- 172.21.15.5:0/62795345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f262c13cd80 msgr2=0x7f262c13f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:55.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 --2- 172.21.15.5:0/62795345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f262c13cd80 0x7f262c13f170 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2624009e50 tx=0x7f262401c8c0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 -- 172.21.15.5:0/62795345 shutdown_connections 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 --2- 172.21.15.5:0/62795345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f262c13cd80 0x7f262c13f170 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 --2- 172.21.15.5:0/62795345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 0x7f262c09cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 -- 172.21.15.5:0/62795345 >> 172.21.15.5:0/62795345 conn(0x7f262c0a6c70 msgr2=0x7f262c0a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 -- 172.21.15.5:0/62795345 shutdown_connections 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 -- 172.21.15.5:0/62795345 wait complete. 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient: build_initial_monmap 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient: monmap: 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:39:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 Processor -- start 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 -- start start 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient: init 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 0x7f262c136340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient(hunting): picked mon.noname-c con 0x7f262c137930 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f262c137d10 0x7f262c136880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:55.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient(hunting): picked mon.noname-b con 0x7f262c137d10 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:39:55.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f262c13cd80 0x7f262c139a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:55.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient(hunting): picked mon.noname-a con 0x7f262c13cd80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:39:55.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:55.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f262c142260 con 0x7f262c13cd80 2024-09-13T11:39:55.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:55.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1420e0 con 0x7f262c137d10 2024-09-13T11:39:55.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:55.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f262c1423e0 con 0x7f262c137930 2024-09-13T11:39:55.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:55.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient(hunting): authenticate will time out at 998.622517s 2024-09-13T11:39:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 0x7f262c136340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262b7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f262c13cd80 0x7f262c139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 0x7f262c136340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:43020/0 (socket says 172.21.15.5:43020) 2024-09-13T11:39:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262b7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f262c13cd80 0x7f262c139a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47056/0 (socket says 172.21.15.5:47056) 2024-09-13T11:39:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 1 -- 172.21.15.5:0/2438398862 learned_addr learned my addr 172.21.15.5:0/2438398862 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:39:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 10 monclient(hunting): get_auth_request con 0x7f262c137930 auth_method 0 2024-09-13T11:39:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262b7fe640 10 monclient(hunting): get_auth_request con 0x7f262c13cd80 auth_method 0 2024-09-13T11:39:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262a7fc640 1 --2- 172.21.15.5:0/2438398862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f262c137d10 0x7f262c136880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262a7fc640 10 monclient(hunting): get_auth_request con 0x7f262c137d10 auth_method 0 2024-09-13T11:39:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 10 monclient(hunting): handle_auth_done global_id 4142 payload 824 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 1 -- 172.21.15.5:0/2438398862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f262c137d10 msgr2=0x7f262c136880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 1 --2- 172.21.15.5:0/2438398862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f262c137d10 0x7f262c136880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 1 -- 172.21.15.5:0/2438398862 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f262c13cd80 msgr2=0x7f262c139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 1 --2- 172.21.15.5:0/2438398862 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f262c13cd80 0x7f262c139a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 10 monclient: _finish_hunting 0 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 1 monclient: found mon.noname-c 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:55.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 1 -- 172.21.15.5:0/2438398862 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f262c120f20 con 0x7f262c137930 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262b7fe640 1 --2- 172.21.15.5:0/2438398862 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f262c13cd80 0x7f262c139a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f262affd640 1 --2- 172.21.15.5:0/2438398862 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 0x7f262c136340 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f261c00e930 tx=0x7f261c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 1 -- 172.21.15.5:0/2438398862 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c017070 con 0x7f262c137930 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 10 monclient: dump: 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 10 monclient: _finish_auth 0 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 10 monclient: _check_auth_tickets 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 1 -- 172.21.15.5:0/2438398862 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f261c00ca00 con 0x7f262c137930 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 5 monclient: authenticate success, global_id 4142 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient: _renew_subs 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 -- 172.21.15.5:0/2438398862 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f262c121200 con 0x7f262c137930 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 1 -- 172.21.15.5:0/2438398862 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c00cdb0 con 0x7f262c137930 2024-09-13T11:39:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 10 monclient: dump: 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26297fa640 4 set_mon_vals no callback set 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient: _renew_subs 2024-09-13T11:39:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26315f5640 1 -- 172.21.15.5:0/2438398862 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f262c13c290 con 0x7f262c137930 2024-09-13T11:39:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 1 -- 172.21.15.5:0/2438398862 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f261c01a070 con 0x7f262c137930 2024-09-13T11:39:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.271+0000 7f26137fe640 1 -- 172.21.15.5:0/2438398862 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f261c0066b0 con 0x7f262c137930 2024-09-13T11:39:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.275+0000 7f26315f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.275+0000 7f26315f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.275+0000 7f26315f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.275+0000 7f26315f5640 1 -- 172.21.15.5:0/2438398862 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f262c09cc90 con 0x7f262c137930 2024-09-13T11:39:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.275+0000 7f26137fe640 1 -- 172.21.15.5:0/2438398862 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f261c006200 con 0x7f262c137930 2024-09-13T11:39:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.275+0000 7f26137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.275+0000 7f26137fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:39:55.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.335+0000 7f26315f5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:39:55.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.335+0000 7f26315f5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:39:55.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.335+0000 7f26315f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:55.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.335+0000 7f26315f5640 1 -- 172.21.15.5:0/2438398862 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f262c06cbe0 con 0x7f262c137930 2024-09-13T11:39:55.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.335+0000 7f26137fe640 1 -- 172.21.15.5:0/2438398862 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f261c0059a0 con 0x7f262c137930 2024-09-13T11:39:55.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.335+0000 7f26137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:39:55.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.335+0000 7f26137fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:39:55.339 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:39:55.339 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":8,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","created":"2024-09-13T11:39:53.326128+0000","modified":"2024-09-13T11:39:55.104089+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T11:39:55.104089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0803d65c-46ec-49b5-a45e-46bcf59ae3f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd592a78-172c-4e1c-9f3e-c49e3e3304d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3cec868-fd4e-472c-83b8-bc2e466671ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"748f14f6-a1d6-4669-9f81-6b1cd7cd5a60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f3a05376-7e83-46f9-b3e6-97b31a0942a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e3b64b2d-c3e5-45d1-90a9-225892061800","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:39:55.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.339+0000 7f26315f5640 10 monclient: shutdown 2024-09-13T11:39:55.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.339+0000 7f26315f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:55.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.339+0000 7f26315f5640 1 -- 172.21.15.5:0/2438398862 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 msgr2=0x7f262c136340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:55.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.339+0000 7f26315f5640 1 --2- 172.21.15.5:0/2438398862 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 0x7f262c136340 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f261c00e930 tx=0x7f261c00edf0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.339+0000 7f26315f5640 1 -- 172.21.15.5:0/2438398862 shutdown_connections 2024-09-13T11:39:55.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.339+0000 7f26315f5640 1 --2- 172.21.15.5:0/2438398862 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f262c13cd80 0x7f262c139a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.339+0000 7f26315f5640 1 --2- 172.21.15.5:0/2438398862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f262c137d10 0x7f262c136880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.339+0000 7f26315f5640 1 --2- 172.21.15.5:0/2438398862 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f262c137930 0x7f262c136340 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:55.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.339+0000 7f26315f5640 1 -- 172.21.15.5:0/2438398862 >> 172.21.15.5:0/2438398862 conn(0x7f262c0a6c70 msgr2=0x7f262c13d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:55.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.339+0000 7f26315f5640 1 -- 172.21.15.5:0/2438398862 shutdown_connections 2024-09-13T11:39:55.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:39:55.339+0000 7f26315f5640 1 -- 172.21.15.5:0/2438398862 wait complete. 2024-09-13T11:39:55.357 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T11:39:55.358 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T11:39:55.963 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T11:39:56.089 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.087+0000 7f0a2e2f1640 10 monclient: get_monmap_and_config 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.087+0000 7f0a2e2f1640 10 monclient: build_initial_monmap 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.087+0000 7f0a2e2f1640 10 monclient: monmap: 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:56.091458+0000 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:56.091458+0000 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:39:56.090 INFO:teuthology.misc.health.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:39:56.091 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:39:56.093 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.087+0000 7f0a2e2f1640 1 Processor -- start 2024-09-13T11:39:56.093 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.087+0000 7f0a2e2f1640 1 -- start start 2024-09-13T11:39:56.093 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.087+0000 7f0a2e2f1640 10 monclient: init 2024-09-13T11:39:56.093 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:56.093 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-13T11:39:56.094 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a28074520 addr v1:172.21.15.39:6789/0 2024-09-13T11:39:56.094 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0a280a5390 0x7f0a2813ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 10 monclient(hunting): picked mon.noname-c con 0x7f0a280a5390 addr v2:172.21.15.39:3300/0 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0a280a57a0 0x7f0a2813f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 10 monclient(hunting): picked mon.noname-a con 0x7f0a280a57a0 addr v2:172.21.15.5:3300/0 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a280a2420 con 0x7f0a28074520 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f0a28074900 con 0x7f0a280a57a0 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f0a280a2140 con 0x7f0a280a5390 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 10 monclient(hunting): authenticate will time out at 999.440333s 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0a280a57a0 0x7f0a2813f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0a280a57a0 0x7f0a2813f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47070/0 (socket says 172.21.15.5:47070) 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 1 -- 172.21.15.5:0/612052978 learned_addr learned my addr 172.21.15.5:0/612052978 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:39:56.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 10 monclient(hunting): get_auth_request con 0x7f0a280a57a0 auth_method 0 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a277fe640 1 --2- 172.21.15.5:0/612052978 >> v2:172.21.15.39:3300/0 conn(0x7f0a280a5390 0x7f0a2813ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a277fe640 10 monclient(hunting): get_auth_request con 0x7f0a280a5390 auth_method 0 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a277fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:56.096 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:56.097 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:56.097 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 10 monclient(hunting): handle_auth_done global_id 4161 payload 274 2024-09-13T11:39:56.097 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 1 -- 172.21.15.5:0/612052978 >> v2:172.21.15.39:3300/0 conn(0x7f0a280a5390 msgr2=0x7f0a2813ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:39:56.097 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 1 --2- 172.21.15.5:0/612052978 >> v2:172.21.15.39:3300/0 conn(0x7f0a280a5390 0x7f0a2813ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:56.097 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 1 -- 172.21.15.5:0/612052978 >> v1:172.21.15.39:6789/0 conn(0x7f0a28074520 legacy=0x7f0a280a4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:56.097 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 10 monclient: _finish_hunting 0 2024-09-13T11:39:56.097 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 1 monclient: found mon.noname-a 2024-09-13T11:39:56.097 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 1 -- 172.21.15.5:0/612052978 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3926867331 0 0) 0x7f0a280a2420 con 0x7f0a28074520 2024-09-13T11:39:56.097 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:56.097 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:39:56.097 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 1 -- 172.21.15.5:0/612052978 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a280a6040 con 0x7f0a280a57a0 2024-09-13T11:39:56.097 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:39:56.103 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2c88e640 1 --2- 172.21.15.5:0/612052978 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a280a57a0 0x7f0a2813f3e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0a2806cd50 tx=0x7f0a2001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2174a9ee5edb2d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:56.103 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 1 -- 172.21.15.5:0/612052978 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a200b0070 con 0x7f0a280a57a0 2024-09-13T11:39:56.103 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:56.103 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:39:56.103 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 10 monclient: dump: 2024-09-13T11:39:56.103 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:39:56.103 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:56.103 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:56.103 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:56.103 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:56.104 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:56.104 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:56.104 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:56.104 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:56.104 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:39:56.104 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 10 monclient: _finish_auth 0 2024-09-13T11:39:56.104 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:39:56.104 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:56.104 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 1 -- 172.21.15.5:0/612052978 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a200ab600 con 0x7f0a280a57a0 2024-09-13T11:39:56.104 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 5 monclient: authenticate success, global_id 4161 2024-09-13T11:39:56.104 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:39:56.104 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 10 monclient: get_monmap_and_config success 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 4 set_mon_vals no callback set 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 1 -- 172.21.15.5:0/612052978 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a200ab9d0 con 0x7f0a280a57a0 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a26ffd640 10 monclient: dump: 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:56.105 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 10 monclient: shutdown 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/612052978 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a280a57a0 msgr2=0x7f0a2813f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 --2- 172.21.15.5:0/612052978 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a280a57a0 0x7f0a2813f3e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0a2806cd50 tx=0x7f0a2001c3e0 comp rx=0 tx=0).stop 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/612052978 shutdown_connections 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 --2- 172.21.15.5:0/612052978 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a280a57a0 0x7f0a2813f3e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 --2- 172.21.15.5:0/612052978 >> v2:172.21.15.39:3300/0 conn(0x7f0a280a5390 0x7f0a2813ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/612052978 >> 172.21.15.5:0/612052978 conn(0x7f0a280aaa40 msgr2=0x7f0a280a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/612052978 shutdown_connections 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.091+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/612052978 wait complete. 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient: build_initial_monmap 2024-09-13T11:39:56.106 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient: monmap: 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 1 Processor -- start 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 1 -- start start 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient: init 2024-09-13T11:39:56.107 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient: _reopen_session rank -1 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a28074520 0x7f0a281cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient(hunting): picked mon.noname-b con 0x7f0a28074520 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a280a5390 0x7f0a281cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient(hunting): picked mon.noname-a con 0x7f0a280a5390 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a280a57a0 0x7f0a281d0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient(hunting): picked mon.noname-c con 0x7f0a280a57a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0a28142350 con 0x7f0a280a5390 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0a280a0fe0 con 0x7f0a28074520 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f0a281424d0 con 0x7f0a280a57a0 2024-09-13T11:39:56.108 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient(hunting): _renew_subs 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient(hunting): authenticate will time out at 999.445465s 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a277fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a280a5390 0x7f0a281cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a280a57a0 0x7f0a281d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a277fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a280a5390 0x7f0a281cfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47084/0 (socket says 172.21.15.5:47084) 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a277fe640 1 -- 172.21.15.5:0/2332309757 learned_addr learned my addr 172.21.15.5:0/2332309757 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a277fe640 10 monclient(hunting): get_auth_request con 0x7f0a280a5390 auth_method 0 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a277fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 10 monclient(hunting): get_auth_request con 0x7f0a280a57a0 auth_method 0 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:56.109 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a27fff640 1 --2- 172.21.15.5:0/2332309757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a28074520 0x7f0a281cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a27fff640 10 monclient(hunting): get_auth_request con 0x7f0a28074520 auth_method 0 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a27fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:39:56.110 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 10 monclient(hunting): handle_auth_done global_id 4145 payload 824 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 1 -- 172.21.15.5:0/2332309757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a28074520 msgr2=0x7f0a281cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 1 --2- 172.21.15.5:0/2332309757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a28074520 0x7f0a281cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 1 -- 172.21.15.5:0/2332309757 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a280a5390 msgr2=0x7f0a281cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 1 --2- 172.21.15.5:0/2332309757 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a280a5390 0x7f0a281cfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 10 monclient: _finish_hunting 0 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 1 monclient: found mon.noname-c 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 1 -- 172.21.15.5:0/2332309757 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2813dac0 con 0x7f0a280a57a0 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a277fe640 1 --2- 172.21.15.5:0/2332309757 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a280a5390 0x7f0a281cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2c88e640 1 --2- 172.21.15.5:0/2332309757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a280a57a0 0x7f0a281d0150 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0a200ab000 tx=0x7f0a2001ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 1 -- 172.21.15.5:0/2332309757 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a200ba070 con 0x7f0a280a57a0 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:56.111 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 10 monclient: dump: 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 10 monclient: _finish_auth 0 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:39:56.112 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 1 -- 172.21.15.5:0/2332309757 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a20004220 con 0x7f0a280a57a0 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 1 -- 172.21.15.5:0/2332309757 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a200b0040 con 0x7f0a280a57a0 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 5 monclient: authenticate success, global_id 4145 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient: _renew_subs 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/2332309757 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a280a9660 con 0x7f0a280a57a0 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a25ffb640 4 set_mon_vals no callback set 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 10 monclient: dump: 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:39:56.113 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient: _renew_subs 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/2332309757 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a2813b760 con 0x7f0a280a57a0 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 1 -- 172.21.15.5:0/2332309757 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0a200ab4c0 con 0x7f0a280a57a0 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a24ff9640 1 -- 172.21.15.5:0/2332309757 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f0a200b6420 con 0x7f0a280a57a0 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:56.114 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:56.115 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:56.115 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.095+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/2332309757 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a280a4c80 con 0x7f0a280a57a0 2024-09-13T11:39:56.115 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.099+0000 7f0a24ff9640 1 -- 172.21.15.5:0/2332309757 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f0a200160f0 con 0x7f0a280a57a0 2024-09-13T11:39:56.115 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.099+0000 7f0a24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:39:56.115 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.099+0000 7f0a24ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:39:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:39:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:39:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:39:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/2332309757 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a2809bbe0 con 0x7f0a280a57a0 2024-09-13T11:39:56.161 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a24ff9640 1 -- 172.21.15.5:0/2332309757 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f0a200039b0 con 0x7f0a280a57a0 2024-09-13T11:39:56.161 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:39:56.161 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a24ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:39:56.161 INFO:teuthology.misc.health.smithi005.stdout: 2024-09-13T11:39:56.161 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":8,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","created":"2024-09-13T11:39:53.326128+0000","modified":"2024-09-13T11:39:55.104089+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T11:39:55.104089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0803d65c-46ec-49b5-a45e-46bcf59ae3f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd592a78-172c-4e1c-9f3e-c49e3e3304d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3cec868-fd4e-472c-83b8-bc2e466671ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"748f14f6-a1d6-4669-9f81-6b1cd7cd5a60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f3a05376-7e83-46f9-b3e6-97b31a0942a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e3b64b2d-c3e5-45d1-90a9-225892061800","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:39:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 10 monclient: shutdown 2024-09-13T11:39:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:39:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/2332309757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a280a57a0 msgr2=0x7f0a281d0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:39:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 1 --2- 172.21.15.5:0/2332309757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a280a57a0 0x7f0a281d0150 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0a200ab000 tx=0x7f0a2001ac20 comp rx=0 tx=0).stop 2024-09-13T11:39:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/2332309757 shutdown_connections 2024-09-13T11:39:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 1 --2- 172.21.15.5:0/2332309757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0a280a57a0 0x7f0a281d0150 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 1 --2- 172.21.15.5:0/2332309757 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a280a5390 0x7f0a281cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 1 --2- 172.21.15.5:0/2332309757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a28074520 0x7f0a281cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:39:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/2332309757 >> 172.21.15.5:0/2332309757 conn(0x7f0a280aaa40 msgr2=0x7f0a280a8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:39:56.163 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/2332309757 shutdown_connections 2024-09-13T11:39:56.163 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:39:56.159+0000 7f0a2e2f1640 1 -- 172.21.15.5:0/2332309757 wait complete. 2024-09-13T11:39:56.177 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-13T11:39:56.280 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-09-13T11:39:56.279+0000 7f477baee7c0 -1 Falling back to public interface 2024-09-13T11:39:56.288 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T11:39:56.287+0000 7f03ddb2a7c0 -1 Falling back to public interface 2024-09-13T11:39:56.346 INFO:tasks.ceph.mgr.x.smithi039.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-13T11:39:56.346 INFO:tasks.ceph.mgr.x.smithi039.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-13T11:39:56.346 INFO:tasks.ceph.mgr.x.smithi039.stderr: from numpy import show_config as show_numpy_config 2024-09-13T11:39:56.502 INFO:tasks.ceph.osd.4.smithi039.stderr:2024-09-13T11:39:56.502+0000 7f8ea871f7c0 -1 Falling back to public interface 2024-09-13T11:39:56.753 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T11:39:56.751+0000 7f52888f27c0 -1 Falling back to public interface 2024-09-13T11:39:56.754 INFO:tasks.ceph.osd.3.smithi039.stderr:2024-09-13T11:39:56.754+0000 7f88771aa7c0 -1 Falling back to public interface 2024-09-13T11:39:56.998 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-13T11:39:56.998+0000 7f5b696307c0 -1 Falling back to public interface 2024-09-13T11:39:59.307 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-13T11:39:59.306+0000 7f65891e0640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T11:39:59.308 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-13T11:39:59.306+0000 7f65891e0640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T11:40:00.246 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-09-13T11:40:00.243+0000 7f477baee7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T11:40:00.270 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T11:40:00.267+0000 7f03ddb2a7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T11:40:00.576 INFO:tasks.ceph.osd.4.smithi039.stderr:2024-09-13T11:40:00.578+0000 7f8ea871f7c0 -1 osd.4 0 log_to_monitors true 2024-09-13T11:40:00.857 INFO:tasks.ceph.osd.3.smithi039.stderr:2024-09-13T11:40:00.858+0000 7f88771aa7c0 -1 osd.3 0 log_to_monitors true 2024-09-13T11:40:00.910 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T11:40:00.907+0000 7f52888f27c0 -1 osd.1 0 log_to_monitors true 2024-09-13T11:40:01.285 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-13T11:40:01.286+0000 7f5b696307c0 -1 osd.5 0 log_to_monitors true 2024-09-13T11:40:01.365 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T11:40:01.363+0000 7f03d9ad3640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T11:40:01.372 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T11:40:01.371+0000 7f03d590d640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:40:02.049 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-09-13T11:40:02.047+0000 7f4777a97640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T11:40:02.053 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-09-13T11:40:02.051+0000 7f476f8c9640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:40:02.356 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T11:40:02.355+0000 7f528489b640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T11:40:02.363 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T11:40:02.363+0000 7f52806d5640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:40:02.371 INFO:tasks.ceph.osd.3.smithi039.stderr:2024-09-13T11:40:02.370+0000 7f8873153640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T11:40:02.372 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-13T11:40:02.370+0000 7f5b655d9640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T11:40:02.372 INFO:tasks.ceph.osd.4.smithi039.stderr:2024-09-13T11:40:02.370+0000 7f8ea46c8640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T11:40:02.383 INFO:tasks.ceph.osd.4.smithi039.stderr:2024-09-13T11:40:02.382+0000 7f8ea0502640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:40:02.383 INFO:tasks.ceph.osd.3.smithi039.stderr:2024-09-13T11:40:02.382+0000 7f886ef8d640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:40:02.384 INFO:tasks.ceph.osd.5.smithi039.stderr:2024-09-13T11:40:02.386+0000 7f5b61413640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:40:02.782 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T11:40:02.906 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.903+0000 7ff96e089640 10 monclient: get_monmap_and_config 2024-09-13T11:40:02.906 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.903+0000 7ff96e089640 10 monclient: build_initial_monmap 2024-09-13T11:40:02.906 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.903+0000 7ff96e089640 10 monclient: monmap: 2024-09-13T11:40:02.906 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-09-13T11:40:02.906 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:02.906 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:40:02.907801+0000 2024-09-13T11:40:02.906 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:40:02.907801+0000 2024-09-13T11:40:02.906 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:02.906 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:02.906 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:02.906 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:02.907 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:02.907 INFO:teuthology.misc.health.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:02.907 INFO:teuthology.misc.health.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:02.907 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.903+0000 7ff96e089640 1 Processor -- start 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.903+0000 7ff96e089640 1 -- start start 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.903+0000 7ff96e089640 10 monclient: init 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9680a0870 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff96809fb50 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff9680a1360 0x7ff96813e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 10 monclient(hunting): picked mon.noname-c con 0x7ff9680a1360 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9680a2010 con 0x7ff96809fb50 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9680a1d30 con 0x7ff9680a0870 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7ff9680a18f0 con 0x7ff9680a1360 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 10 monclient(hunting): authenticate will time out at 1006.256572s 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff966ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff96809fb50 0x7ff96813ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:53546/0 (socket says 172.21.15.5:53546) 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff966ffd640 1 -- 172.21.15.5:0/3637431568 learned_addr learned my addr 172.21.15.5:0/3637431568 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 1 --2- 172.21.15.5:0/3637431568 >> v2:172.21.15.39:3300/0 conn(0x7ff9680a1360 0x7ff96813e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2640242948 0 0) 0x7ff9680a2010 con 0x7ff96809fb50 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient(hunting): _negotiate 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient(hunting): my global_id is 4197 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff950003530 con 0x7ff96809fb50 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): get_auth_request con 0x7ff9680a1360 auth_method 0 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2935807638 0 0) 0x7ff950003530 con 0x7ff96809fb50 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 >> v2:172.21.15.39:3300/0 conn(0x7ff9680a1360 msgr2=0x7ff96813e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 --2- 172.21.15.5:0/3637431568 >> v2:172.21.15.39:3300/0 conn(0x7ff9680a1360 0x7ff96813e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 >> v1:172.21.15.39:6789/0 conn(0x7ff9680a0870 legacy=0x7ff9680a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient: _finish_hunting 0 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-13T11:40:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96809f5f0 con 0x7ff96809fb50 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3199362888 0 0) 0x7ff95c001700 con 0x7ff96809fb50 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 5 monclient: authenticate success, global_id 4197 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient: dump: 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:02.916 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 >> v1:172.21.15.5:6789/0 conn(0x7ff96809fb50 legacy=0x7ff96813ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 --2- 172.21.15.5:0/3637431568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff950003ab0 0x7ff950003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient(hunting): picked mon.b con 0x7ff950003ab0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 --2- 172.21.15.5:0/3637431568 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff950004810 0x7ff950004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient(hunting): picked mon.c con 0x7ff950004810 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 --2- 172.21.15.5:0/3637431568 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9500053d0 0x7ff95000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient(hunting): picked mon.a con 0x7ff9500053d0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff95000c260 con 0x7ff9500053d0 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff95000c410 con 0x7ff950003ab0 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7ff95000c610 con 0x7ff950004810 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9667fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff96e089640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff966ffd640 1 --2- 172.21.15.5:0/3637431568 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff950004810 0x7ff950004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff966ffd640 10 monclient(hunting): get_auth_request con 0x7ff950004810 auth_method 0 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 1 --2- 172.21.15.5:0/3637431568 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9500053d0 0x7ff95000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff966ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff966ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): get_auth_request con 0x7ff9500053d0 auth_method 0 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff966ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff966ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff966ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:40:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient(hunting): handle_auth_done global_id 4197 payload 293 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 1 -- 172.21.15.5:0/3637431568 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff950004810 msgr2=0x7ff950004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 1 --2- 172.21.15.5:0/3637431568 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff950004810 0x7ff950004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 1 -- 172.21.15.5:0/3637431568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff950003ab0 msgr2=0x7ff950003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9677fe640 1 --2- 172.21.15.5:0/3637431568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff950003ab0 0x7ff950003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 1 --2- 172.21.15.5:0/3637431568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff950003ab0 0x7ff950003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 1 monclient: found mon.a 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 1 -- 172.21.15.5:0/3637431568 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff950003530 con 0x7ff9500053d0 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient: _finish_auth 0 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:02.920 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff967fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9677fe640 10 monclient: get_auth_request con 0x7ff950003ab0 auth_method 0 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.907+0000 7ff9677fe640 1 --2- 172.21.15.5:0/3637431568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff950003ab0 0x7ff950003ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 1 -- 172.21.15.5:0/3637431568 reap_dead start 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 1 --2- 172.21.15.5:0/3637431568 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9500053d0 0x7ff95000b9d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff960009fc0 tx=0x7ff96001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c892c8dc973a4377 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff960009c10 con 0x7ff9500053d0 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9667fc640 10 monclient: dump: 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:02.921 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff960003ba0 con 0x7ff9500053d0 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9667fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 4 set_mon_vals no callback set 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9667fc640 1 -- 172.21.15.5:0/3637431568 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9600a5890 con 0x7ff9500053d0 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9667fc640 10 monclient: dump: 2024-09-13T11:40:02.922 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient: shutdown 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 -- 172.21.15.5:0/3637431568 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9500053d0 msgr2=0x7ff95000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 --2- 172.21.15.5:0/3637431568 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9500053d0 0x7ff95000b9d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff960009fc0 tx=0x7ff96001cca0 comp rx=0 tx=0).stop 2024-09-13T11:40:02.923 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 -- 172.21.15.5:0/3637431568 shutdown_connections 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 --2- 172.21.15.5:0/3637431568 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9500053d0 0x7ff95000b9d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 -- 172.21.15.5:0/3637431568 >> 172.21.15.5:0/3637431568 conn(0x7ff9680a6720 msgr2=0x7ff9680a2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 -- 172.21.15.5:0/3637431568 shutdown_connections 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 -- 172.21.15.5:0/3637431568 wait complete. 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient: build_initial_monmap 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient: monmap: 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:02.924 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:02.925 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:02.925 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:02.925 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:02.925 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 Processor -- start 2024-09-13T11:40:02.925 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 -- start start 2024-09-13T11:40:02.925 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient: init 2024-09-13T11:40:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:40:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9500053d0 0x7ff9681cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient(hunting): picked mon.noname-b con 0x7ff9500053d0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff968131c40 0x7ff9681320c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient(hunting): picked mon.noname-c con 0x7ff968131c40 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9680704a0 0x7ff9680708e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient(hunting): picked mon.noname-a con 0x7ff9680704a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff9680a0fe0 con 0x7ff9680704a0 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff9680a0e60 con 0x7ff9500053d0 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7ff9680a1160 con 0x7ff968131c40 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient(hunting): authenticate will time out at 1006.262778s 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff966ffd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff968131c40 0x7ff9681320c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9680704a0 0x7ff9680708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff966ffd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff968131c40 0x7ff9681320c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:35554/0 (socket says 172.21.15.5:35554) 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff966ffd640 1 -- 172.21.15.5:0/1742338874 learned_addr learned my addr 172.21.15.5:0/1742338874 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff966ffd640 10 monclient(hunting): get_auth_request con 0x7ff968131c40 auth_method 0 2024-09-13T11:40:02.930 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff966ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:02.930 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:02.930 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:02.930 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 10 monclient(hunting): get_auth_request con 0x7ff9680704a0 auth_method 0 2024-09-13T11:40:02.930 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:02.930 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:02.930 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:02.930 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:02.930 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:02.930 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff966ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff966ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff966ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9677fe640 1 --2- 172.21.15.5:0/1742338874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9500053d0 0x7ff9681cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9677fe640 10 monclient(hunting): get_auth_request con 0x7ff9500053d0 auth_method 0 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9677fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff9677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 10 monclient(hunting): handle_auth_done global_id 4200 payload 824 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 1 -- 172.21.15.5:0/1742338874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff968131c40 msgr2=0x7ff9681320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 1 --2- 172.21.15.5:0/1742338874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff968131c40 0x7ff9681320c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 1 -- 172.21.15.5:0/1742338874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9500053d0 msgr2=0x7ff9681cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 1 --2- 172.21.15.5:0/1742338874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9500053d0 0x7ff9681cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:02.931 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 1 monclient: found mon.noname-a 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 1 -- 172.21.15.5:0/1742338874 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9681cf900 con 0x7ff9680704a0 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff967fff640 1 --2- 172.21.15.5:0/1742338874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9680704a0 0x7ff9680708e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff96809ff80 tx=0x7ff960009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 1 -- 172.21.15.5:0/1742338874 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff960009c10 con 0x7ff9680704a0 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 10 monclient: dump: 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:02.932 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 10 monclient: _finish_auth 0 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 1 -- 172.21.15.5:0/1742338874 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9600be070 con 0x7ff9680704a0 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 1 -- 172.21.15.5:0/1742338874 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9600b8690 con 0x7ff9680704a0 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 5 monclient: authenticate success, global_id 4200 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient: _renew_subs 2024-09-13T11:40:02.933 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 -- 172.21.15.5:0/1742338874 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff96809d0d0 con 0x7ff9680704a0 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff94ffff640 10 monclient: dump: 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:02.934 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff965ffb640 4 set_mon_vals no callback set 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient: _renew_subs 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.911+0000 7ff96e089640 1 -- 172.21.15.5:0/1742338874 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff968071f60 con 0x7ff9680704a0 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.915+0000 7ff96e089640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.915+0000 7ff96e089640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.915+0000 7ff96e089640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.915+0000 7ff96e089640 1 -- 172.21.15.5:0/1742338874 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff968135650 con 0x7ff9680704a0 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.915+0000 7ff94ffff640 1 -- 172.21.15.5:0/1742338874 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 4) ==== 48241+0+0 (secure 0 0 0) 0x7ff9600b8830 con 0x7ff9680704a0 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.915+0000 7ff94ffff640 1 --2- 172.21.15.5:0/1742338874 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff95403b240 0x7ff95403d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.915+0000 7ff94ffff640 1 -- 172.21.15.5:0/1742338874 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 2207+0+0 (secure 0 0 0) 0x7ff9600e97a0 con 0x7ff9680704a0 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.915+0000 7ff9677fe640 1 --2- 172.21.15.5:0/1742338874 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff95403b240 0x7ff95403d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.915+0000 7ff9677fe640 10 monclient: get_auth_request con 0x7ff95403b240 auth_method 0 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.915+0000 7ff9677fe640 1 --2- 172.21.15.5:0/1742338874 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff95403b240 0x7ff95403d700 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff958000c10 tx=0x7ff958009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:02.935 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.919+0000 7ff94ffff640 1 -- 172.21.15.5:0/1742338874 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7ff9600169b0 con 0x7ff9680704a0 2024-09-13T11:40:02.936 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.919+0000 7ff94ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:02.936 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:02.919+0000 7ff94ffff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:03.091 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:03.091 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:03.091 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:03.091 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 -- 172.21.15.5:0/1742338874 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9680a58d0 con 0x7ff9680704a0 2024-09-13T11:40:03.092 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff94ffff640 1 -- 172.21.15.5:0/1742338874 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+6081 (secure 0 0 0) 0x7ff9600b8d00 con 0x7ff9680704a0 2024-09-13T11:40:03.092 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff94ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:03.092 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff94ffff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:03.092 INFO:teuthology.misc.health.smithi005.stdout: 2024-09-13T11:40:03.092 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":11,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","created":"2024-09-13T11:39:53.326128+0000","modified":"2024-09-13T11:40:02.346608+0000","last_up_change":"2024-09-13T11:40:02.346608+0000","last_in_change":"2024-09-13T11:39:55.104089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0803d65c-46ec-49b5-a45e-46bcf59ae3f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":4278099835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":4278099835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":4278099835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":4278099835}]},"public_addr":"172.21.15.5:6804/4278099835","cluster_addr":"172.21.15.5:6805/4278099835","heartbeat_back_addr":"172.21.15.5:6807/4278099835","heartbeat_front_addr":"172.21.15.5:6806/4278099835","state":["exists","up"]},{"osd":1,"uuid":"fd592a78-172c-4e1c-9f3e-c49e3e3304d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3cec868-fd4e-472c-83b8-bc2e466671ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":1907498696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":1907498696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":1907498696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":1907498696}]},"public_addr":"172.21.15.5:6800/1907498696","cluster_addr":"172.21.15.5:6801/1907498696","heartbeat_back_addr":"172.21.15.5:6803/1907498696","heartbeat_front_addr":"172.21.15.5:6802/1907498696","state":["exists","up"]},{"osd":3,"uuid":"748f14f6-a1d6-4669-9f81-6b1cd7cd5a60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f3a05376-7e83-46f9-b3e6-97b31a0942a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e3b64b2d-c3e5-45d1-90a9-225892061800","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:40:03.094 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 -- 172.21.15.5:0/1742338874 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff95403b240 msgr2=0x7ff95403d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:03.094 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 --2- 172.21.15.5:0/1742338874 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff95403b240 0x7ff95403d700 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff958000c10 tx=0x7ff958009040 comp rx=0 tx=0).stop 2024-09-13T11:40:03.094 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 10 monclient: shutdown 2024-09-13T11:40:03.094 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:03.094 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 -- 172.21.15.5:0/1742338874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9680704a0 msgr2=0x7ff9680708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:03.094 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 --2- 172.21.15.5:0/1742338874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9680704a0 0x7ff9680708e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff96809ff80 tx=0x7ff960009750 comp rx=0 tx=0).stop 2024-09-13T11:40:03.094 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 -- 172.21.15.5:0/1742338874 shutdown_connections 2024-09-13T11:40:03.094 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 --2- 172.21.15.5:0/1742338874 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff95403b240 0x7ff95403d700 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:03.094 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 --2- 172.21.15.5:0/1742338874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9680704a0 0x7ff9680708e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:03.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 --2- 172.21.15.5:0/1742338874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff968131c40 0x7ff9681320c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:03.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 --2- 172.21.15.5:0/1742338874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9500053d0 0x7ff9681cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:03.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 -- 172.21.15.5:0/1742338874 >> 172.21.15.5:0/1742338874 conn(0x7ff9680a6720 msgr2=0x7ff968132690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:03.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 -- 172.21.15.5:0/1742338874 shutdown_connections 2024-09-13T11:40:03.095 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:03.091+0000 7ff96e089640 1 -- 172.21.15.5:0/1742338874 wait complete. 2024-09-13T11:40:03.109 DEBUG:teuthology.misc:2 of 6 OSDs are up 2024-09-13T11:40:09.718 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T11:40:09.844 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient: get_monmap_and_config 2024-09-13T11:40:09.845 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient: build_initial_monmap 2024-09-13T11:40:09.845 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient: monmap: 2024-09-13T11:40:09.845 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-09-13T11:40:09.845 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:09.845 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:40:09.846123+0000 2024-09-13T11:40:09.845 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:40:09.846123+0000 2024-09-13T11:40:09.845 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:09.845 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:09.846 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:09.846 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:09.846 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:09.846 INFO:teuthology.misc.health.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:09.846 INFO:teuthology.misc.health.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:09.846 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:09.848 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 1 Processor -- start 2024-09-13T11:40:09.849 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 1 -- start start 2024-09-13T11:40:09.849 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient: init 2024-09-13T11:40:09.849 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:09.849 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-13T11:40:09.849 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c074520 0x7ff27c0a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:09.849 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient(hunting): picked mon.noname-b con 0x7ff27c074520 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:09.849 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff27c0a51c0 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:09.849 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff27c0a55c0 0x7ff27c138150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient(hunting): picked mon.noname-a con 0x7ff27c0a55c0 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff27c0a2420 con 0x7ff27c0a51c0 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff27c074900 con 0x7ff27c0a55c0 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7ff27c0a2140 con 0x7ff27c074520 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff283840640 10 monclient(hunting): authenticate will time out at 1013.195576s 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff281ddd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff27c0a55c0 0x7ff27c138150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff2815dc640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c074520 0x7ff27c0a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff2815dc640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c074520 0x7ff27c0a4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:58994/0 (socket says 172.21.15.5:58994) 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff2815dc640 1 -- 172.21.15.5:0/2852337299 learned_addr learned my addr 172.21.15.5:0/2852337299 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:09.850 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff281ddd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff27c0a55c0 0x7ff27c138150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57080/0 (socket says 172.21.15.5:57080) 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff2815dc640 10 monclient(hunting): get_auth_request con 0x7ff27c074520 auth_method 0 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff2815dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff2815dc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff2815dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff281ddd640 10 monclient(hunting): get_auth_request con 0x7ff27c0a55c0 auth_method 0 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff281ddd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff281ddd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff281ddd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff281ddd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff281ddd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff281ddd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff2815dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:09.851 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.843+0000 7ff2815dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:09.852 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff2815dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:09.852 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff281ddd640 10 monclient(hunting): handle_auth_done global_id 4209 payload 274 2024-09-13T11:40:09.852 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff281ddd640 1 -- 172.21.15.5:0/2852337299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c074520 msgr2=0x7ff27c0a4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:09.852 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff281ddd640 1 --2- 172.21.15.5:0/2852337299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c074520 0x7ff27c0a4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:09.852 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff281ddd640 1 -- 172.21.15.5:0/2852337299 >> v1:172.21.15.39:6789/0 conn(0x7ff27c0a51c0 legacy=0x7ff27c137a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:09.852 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff281ddd640 10 monclient: _finish_hunting 0 2024-09-13T11:40:09.852 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff281ddd640 1 monclient: found mon.noname-a 2024-09-13T11:40:09.852 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff281ddd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:09.852 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff281ddd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:09.852 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff281ddd640 1 -- 172.21.15.5:0/2852337299 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff27c0a6040 con 0x7ff27c0a55c0 2024-09-13T11:40:09.852 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff2815dc640 1 --2- 172.21.15.5:0/2852337299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c074520 0x7ff27c0a4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:09.852 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff281ddd640 1 --2- 172.21.15.5:0/2852337299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff27c0a55c0 0x7ff27c138150 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff274009dc0 tx=0x7ff27401c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15bc08f100ee409c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:09.858 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 1 -- 172.21.15.5:0/2852337299 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740ac070 con 0x7ff27c0a55c0 2024-09-13T11:40:09.858 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:09.858 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:09.858 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 10 monclient: dump: 2024-09-13T11:40:09.858 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:40:09.858 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:09.858 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:09.858 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:09.858 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:09.858 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:09.858 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 10 monclient: _finish_auth 0 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 1 -- 172.21.15.5:0/2852337299 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2740a6660 con 0x7ff27c0a55c0 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 1 -- 172.21.15.5:0/2852337299 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740a6a30 con 0x7ff27c0a55c0 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 5 monclient: authenticate success, global_id 4209 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 4 set_mon_vals no callback set 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:09.859 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff273fff640 10 monclient: dump: 2024-09-13T11:40:09.860 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:40:09.860 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:09.860 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:09.860 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:09.860 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:09.860 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:09.860 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:09.860 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:09.860 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 10 monclient: shutdown 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 1 -- 172.21.15.5:0/2852337299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff27c0a55c0 msgr2=0x7ff27c138150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 1 --2- 172.21.15.5:0/2852337299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff27c0a55c0 0x7ff27c138150 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff274009dc0 tx=0x7ff27401c810 comp rx=0 tx=0).stop 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 1 -- 172.21.15.5:0/2852337299 shutdown_connections 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 1 --2- 172.21.15.5:0/2852337299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff27c0a55c0 0x7ff27c138150 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 1 --2- 172.21.15.5:0/2852337299 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c074520 0x7ff27c0a4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 1 -- 172.21.15.5:0/2852337299 >> 172.21.15.5:0/2852337299 conn(0x7ff27c0aaa40 msgr2=0x7ff27c0a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 1 -- 172.21.15.5:0/2852337299 shutdown_connections 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 1 -- 172.21.15.5:0/2852337299 wait complete. 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 10 monclient: build_initial_monmap 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.847+0000 7ff283840640 10 monclient: monmap: 2024-09-13T11:40:09.861 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 1 Processor -- start 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 1 -- start start 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient: init 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:09.862 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff27c074520 0x7ff27c1377d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient(hunting): picked mon.noname-a con 0x7ff27c074520 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff27c0a51c0 0x7ff27c13d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient(hunting): picked mon.noname-b con 0x7ff27c0a51c0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c0a55c0 0x7ff27c13df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient(hunting): picked mon.noname-c con 0x7ff27c0a55c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c138690 con 0x7ff27c074520 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c0a1010 con 0x7ff27c0a51c0 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7ff27c138810 con 0x7ff27c0a55c0 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:09.863 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient(hunting): authenticate will time out at 1013.200724s 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c0a55c0 0x7ff27c13df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff2815dc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff27c074520 0x7ff27c1377d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c0a55c0 0x7ff27c13df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59008/0 (socket says 172.21.15.5:59008) 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 1 -- 172.21.15.5:0/3335020331 learned_addr learned my addr 172.21.15.5:0/3335020331 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 10 monclient(hunting): get_auth_request con 0x7ff27c0a55c0 auth_method 0 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff2815dc640 10 monclient(hunting): get_auth_request con 0x7ff27c074520 auth_method 0 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff2815dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff2815dc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff2815dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:09.864 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff2815dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff2815dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff2815dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff280ddb640 1 --2- 172.21.15.5:0/3335020331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff27c0a51c0 0x7ff27c13d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 1 -- 172.21.15.5:0/3335020331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff27c0a51c0 msgr2=0x7ff27c13d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 1 --2- 172.21.15.5:0/3335020331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff27c0a51c0 0x7ff27c13d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 1 -- 172.21.15.5:0/3335020331 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff27c074520 msgr2=0x7ff27c1377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 1 --2- 172.21.15.5:0/3335020331 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff27c074520 0x7ff27c1377d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 10 monclient: _finish_hunting 0 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 1 monclient: found mon.noname-c 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:09.865 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 1 -- 172.21.15.5:0/3335020331 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff27c13f3a0 con 0x7ff27c0a55c0 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff280ddb640 10 monclient: get_auth_request con 0x7ff27c0a51c0 auth_method 0 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff2815dc640 1 --2- 172.21.15.5:0/3335020331 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff27c074520 0x7ff27c1377d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff280ddb640 1 --2- 172.21.15.5:0/3335020331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff27c0a51c0 0x7ff27c13d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff281ddd640 1 --2- 172.21.15.5:0/3335020331 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c0a55c0 0x7ff27c13df30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff27401c640 tx=0x7ff2740195b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 1 -- 172.21.15.5:0/3335020331 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740c0070 con 0x7ff27c0a55c0 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 10 monclient: dump: 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:09.866 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 5 monclient: authenticate success, global_id 4196 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 1 -- 172.21.15.5:0/3335020331 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2740a6e50 con 0x7ff27c0a55c0 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient: _renew_subs 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 1 -- 172.21.15.5:0/3335020331 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff27c0a9a30 con 0x7ff27c0a55c0 2024-09-13T11:40:09.867 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:09.868 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 1 -- 172.21.15.5:0/3335020331 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740ac050 con 0x7ff27c0a55c0 2024-09-13T11:40:09.868 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:09.868 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:09.868 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 10 monclient: dump: 2024-09-13T11:40:09.868 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-13T11:40:09.868 INFO:teuthology.misc.health.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:09.868 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:09.868 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:09.868 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:09.868 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:09.869 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:09.869 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:09.869 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:09.869 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-13T11:40:09.869 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff272ffd640 4 set_mon_vals no callback set 2024-09-13T11:40:09.869 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient: _renew_subs 2024-09-13T11:40:09.869 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:09.869 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 1 -- 172.21.15.5:0/3335020331 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff27c122180 con 0x7ff27c0a55c0 2024-09-13T11:40:09.869 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:09.869 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:09.870 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:09.870 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff283840640 1 -- 172.21.15.5:0/3335020331 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff27c0a4c80 con 0x7ff27c0a55c0 2024-09-13T11:40:09.870 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 1 -- 172.21.15.5:0/3335020331 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7ff2740c3070 con 0x7ff27c0a55c0 2024-09-13T11:40:09.870 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 1 --2- 172.21.15.5:0/3335020331 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff26003b2a0 0x7ff26003d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:09.870 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff271ffb640 1 -- 172.21.15.5:0/3335020331 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (secure 0 0 0) 0x7ff2740e94d0 con 0x7ff27c0a55c0 2024-09-13T11:40:09.870 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff2815dc640 1 --2- 172.21.15.5:0/3335020331 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff26003b2a0 0x7ff26003d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:09.870 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.851+0000 7ff2815dc640 10 monclient: get_auth_request con 0x7ff26003b2a0 auth_method 0 2024-09-13T11:40:09.870 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.855+0000 7ff2815dc640 1 --2- 172.21.15.5:0/3335020331 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff26003b2a0 0x7ff26003d760 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff2780096f0 tx=0x7ff278009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:09.870 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.859+0000 7ff271ffb640 1 -- 172.21.15.5:0/3335020331 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7ff274016520 con 0x7ff27c0a55c0 2024-09-13T11:40:09.870 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.859+0000 7ff271ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:09.871 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:09.859+0000 7ff271ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:10.026 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.023+0000 7ff283840640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:10.027 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.023+0000 7ff283840640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:10.027 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.023+0000 7ff283840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:10.027 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.023+0000 7ff283840640 1 -- 172.21.15.5:0/3335020331 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff27c13a580 con 0x7ff27c0a55c0 2024-09-13T11:40:10.027 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff271ffb640 1 -- 172.21.15.5:0/3335020331 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+9009 (secure 0 0 0) 0x7ff2740a7440 con 0x7ff27c0a55c0 2024-09-13T11:40:10.027 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff271ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:10.027 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff271ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:10.028 INFO:teuthology.misc.health.smithi005.stdout: 2024-09-13T11:40:10.028 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":15,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","created":"2024-09-13T11:39:53.326128+0000","modified":"2024-09-13T11:40:06.385011+0000","last_up_change":"2024-09-13T11:40:03.374234+0000","last_in_change":"2024-09-13T11:39:55.104089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:40:04.344231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0803d65c-46ec-49b5-a45e-46bcf59ae3f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":4278099835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":4278099835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":4278099835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":4278099835}]},"public_addr":"172.21.15.5:6804/4278099835","cluster_addr":"172.21.15.5:6805/4278099835","heartbeat_back_addr":"172.21.15.5:6807/4278099835","heartbeat_front_addr":"172.21.15.5:6806/4278099835","state":["exists","up"]},{"osd":1,"uuid":"fd592a78-172c-4e1c-9f3e-c49e3e3304d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":1447734001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":1447734001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":1447734001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":1447734001}]},"public_addr":"172.21.15.5:6808/1447734001","cluster_addr":"172.21.15.5:6809/1447734001","heartbeat_back_addr":"172.21.15.5:6811/1447734001","heartbeat_front_addr":"172.21.15.5:6810/1447734001","state":["exists","up"]},{"osd":2,"uuid":"c3cec868-fd4e-472c-83b8-bc2e466671ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":1907498696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":1907498696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":1907498696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":1907498696}]},"public_addr":"172.21.15.5:6800/1907498696","cluster_addr":"172.21.15.5:6801/1907498696","heartbeat_back_addr":"172.21.15.5:6803/1907498696","heartbeat_front_addr":"172.21.15.5:6802/1907498696","state":["exists","up"]},{"osd":3,"uuid":"748f14f6-a1d6-4669-9f81-6b1cd7cd5a60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3188120659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":3188120659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":3188120659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":3188120659}]},"public_addr":"172.21.15.39:6804/3188120659","cluster_addr":"172.21.15.39:6805/3188120659","heartbeat_back_addr":"172.21.15.39:6807/3188120659","heartbeat_front_addr":"172.21.15.39:6806/3188120659","state":["exists","up"]},{"osd":4,"uuid":"f3a05376-7e83-46f9-b3e6-97b31a0942a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":4264535364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":4264535364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":4264535364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4264535364}]},"public_addr":"172.21.15.39:6800/4264535364","cluster_addr":"172.21.15.39:6801/4264535364","heartbeat_back_addr":"172.21.15.39:6803/4264535364","heartbeat_front_addr":"172.21.15.39:6802/4264535364","state":["exists","up"]},{"osd":5,"uuid":"e3b64b2d-c3e5-45d1-90a9-225892061800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1185247880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":1185247880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":1185247880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1185247880}]},"public_addr":"172.21.15.39:6808/1185247880","cluster_addr":"172.21.15.39:6809/1185247880","heartbeat_back_addr":"172.21.15.39:6811/1185247880","heartbeat_front_addr":"172.21.15.39:6810/1185247880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.244929+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.890770+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.206643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.846803+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.612582+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:02.321712+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:40:10.029 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 1 -- 172.21.15.5:0/3335020331 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff26003b2a0 msgr2=0x7ff26003d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:10.029 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 1 --2- 172.21.15.5:0/3335020331 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff26003b2a0 0x7ff26003d760 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff2780096f0 tx=0x7ff278009240 comp rx=0 tx=0).stop 2024-09-13T11:40:10.030 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 10 monclient: shutdown 2024-09-13T11:40:10.030 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:10.030 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 1 -- 172.21.15.5:0/3335020331 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c0a55c0 msgr2=0x7ff27c13df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:10.030 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 1 --2- 172.21.15.5:0/3335020331 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c0a55c0 0x7ff27c13df30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff27401c640 tx=0x7ff2740195b0 comp rx=0 tx=0).stop 2024-09-13T11:40:10.030 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 1 -- 172.21.15.5:0/3335020331 shutdown_connections 2024-09-13T11:40:10.030 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 1 --2- 172.21.15.5:0/3335020331 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff26003b2a0 0x7ff26003d760 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:10.030 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 1 --2- 172.21.15.5:0/3335020331 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff27c0a55c0 0x7ff27c13df30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:10.030 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 1 --2- 172.21.15.5:0/3335020331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff27c0a51c0 0x7ff27c13d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:10.030 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 1 --2- 172.21.15.5:0/3335020331 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff27c074520 0x7ff27c1377d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:10.030 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 1 -- 172.21.15.5:0/3335020331 >> 172.21.15.5:0/3335020331 conn(0x7ff27c0aaa40 msgr2=0x7ff27c0a89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:10.030 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 1 -- 172.21.15.5:0/3335020331 shutdown_connections 2024-09-13T11:40:10.030 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T11:40:10.027+0000 7ff283840640 1 -- 172.21.15.5:0/3335020331 wait complete. 2024-09-13T11:40:10.044 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-13T11:40:10.044 INFO:tasks.ceph:Creating RBD pool 2024-09-13T11:40:10.044 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-13T11:40:10.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient: get_monmap_and_config 2024-09-13T11:40:10.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient: build_initial_monmap 2024-09-13T11:40:10.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient: monmap: 2024-09-13T11:40:10.167 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:10.167 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:10.167 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:10.168926+0000 2024-09-13T11:40:10.167 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:10.168926+0000 2024-09-13T11:40:10.167 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:10.167 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:10.168 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:10.168 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:10.168 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:10.168 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:10.168 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:10.168 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:10.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 1 Processor -- start 2024-09-13T11:40:10.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 1 -- start start 2024-09-13T11:40:10.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient: init 2024-09-13T11:40:10.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:10.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541006fe40 0x7f54100a3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient(hunting): picked mon.noname-b con 0x7f541006fe40 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f54100a3e60 0x7f54100a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient(hunting): picked mon.noname-c con 0x7f54100a3e60 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54100a5020 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54100a6000 con 0x7f54100a5020 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f5410070220 con 0x7f54100a3e60 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f54100a1b70 con 0x7f541006fe40 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f5414cb8640 10 monclient(hunting): authenticate will time out at 1013.517677s 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540ed9d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f54100a5020 0x7f541013ec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36760/0 (socket says 172.21.15.5:36760) 2024-09-13T11:40:10.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541006fe40 0x7f54100a3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540ed9d640 1 -- 172.21.15.5:0/3313239348 learned_addr learned my addr 172.21.15.5:0/3313239348 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 10 monclient(hunting): get_auth_request con 0x7f541006fe40 auth_method 0 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540dd9b640 1 --2- 172.21.15.5:0/3313239348 >> v2:172.21.15.39:3300/0 conn(0x7f54100a3e60 0x7f54100a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540d59a640 1 -- 172.21.15.5:0/3313239348 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692145179 0 0) 0x7f54100a6000 con 0x7f54100a5020 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540d59a640 10 monclient(hunting): _negotiate 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540d59a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:10.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540d59a640 10 monclient(hunting): my global_id is 4215 2024-09-13T11:40:10.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540d59a640 1 -- 172.21.15.5:0/3313239348 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f0003530 con 0x7f54100a5020 2024-09-13T11:40:10.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 10 monclient(hunting): handle_auth_done global_id 4199 payload 274 2024-09-13T11:40:10.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 1 -- 172.21.15.5:0/3313239348 >> v2:172.21.15.39:3300/0 conn(0x7f54100a3e60 msgr2=0x7f54100a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:10.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 1 --2- 172.21.15.5:0/3313239348 >> v2:172.21.15.39:3300/0 conn(0x7f54100a3e60 0x7f54100a4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:10.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 1 -- 172.21.15.5:0/3313239348 >> v1:172.21.15.5:6789/0 conn(0x7f54100a5020 legacy=0x7f541013ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:10.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 10 monclient: _finish_hunting 0 2024-09-13T11:40:10.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 1 monclient: found mon.noname-b 2024-09-13T11:40:10.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:10.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:40:10.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540e59c640 1 -- 172.21.15.5:0/3313239348 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54100a4570 con 0x7f541006fe40 2024-09-13T11:40:10.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540dd9b640 10 monclient: get_auth_request con 0x7f54100a3e60 auth_method 0 2024-09-13T11:40:10.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.167+0000 7f540dd9b640 1 --2- 172.21.15.5:0/3313239348 >> v2:172.21.15.39:3300/0 conn(0x7f54100a3e60 0x7f54100a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:10.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540e59c640 1 --2- 172.21.15.5:0/3313239348 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541006fe40 0x7f54100a3920 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f53f80097c0 tx=0x7f53f801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d124491d85eaa20b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:10.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 1 -- 172.21.15.5:0/3313239348 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f80b0070 con 0x7f541006fe40 2024-09-13T11:40:10.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:10.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:10.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 10 monclient: dump: 2024-09-13T11:40:10.179 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:10.179 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:10.179 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:10.179 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 10 monclient: _finish_auth 0 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 10 monclient: _check_auth_tickets 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 1 -- 172.21.15.5:0/3313239348 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53f80ab600 con 0x7f541006fe40 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 1 -- 172.21.15.5:0/3313239348 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f80ab9d0 con 0x7f541006fe40 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:10.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540d59a640 10 monclient: dump: 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 5 monclient: authenticate success, global_id 4199 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:10.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 4 set_mon_vals no callback set 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient: shutdown 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 -- 172.21.15.5:0/3313239348 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541006fe40 msgr2=0x7f54100a3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 --2- 172.21.15.5:0/3313239348 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541006fe40 0x7f54100a3920 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f53f80097c0 tx=0x7f53f801c360 comp rx=0 tx=0).stop 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 -- 172.21.15.5:0/3313239348 shutdown_connections 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 --2- 172.21.15.5:0/3313239348 >> v2:172.21.15.39:3300/0 conn(0x7f54100a3e60 0x7f54100a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 --2- 172.21.15.5:0/3313239348 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541006fe40 0x7f54100a3920 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 -- 172.21.15.5:0/3313239348 >> 172.21.15.5:0/3313239348 conn(0x7f54100a6be0 msgr2=0x7f54100a6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 -- 172.21.15.5:0/3313239348 shutdown_connections 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 -- 172.21.15.5:0/3313239348 wait complete. 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient: build_initial_monmap 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient: monmap: 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:10.182 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 Processor -- start 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 -- start start 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient: init 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:10.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:40:10.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541006fe40 0x7f54101cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:10.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient(hunting): picked mon.noname-c con 0x7f541006fe40 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:10.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54100a3e60 0x7f54101d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:10.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient(hunting): picked mon.noname-a con 0x7f54100a3e60 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:10.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54100a5020 0x7f541013c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:10.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient(hunting): picked mon.noname-b con 0x7f54100a5020 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:10.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:10.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f54100a4240 con 0x7f54100a3e60 2024-09-13T11:40:10.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f54100a15e0 con 0x7f54100a5020 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f54100a9060 con 0x7f541006fe40 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient(hunting): authenticate will time out at 1013.522719s 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540e59c640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541006fe40 0x7f54101cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54100a3e60 0x7f54101d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540e59c640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541006fe40 0x7f54101cfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59026/0 (socket says 172.21.15.5:59026) 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54100a3e60 0x7f54101d01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57100/0 (socket says 172.21.15.5:57100) 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540e59c640 1 -- 172.21.15.5:0/4276894903 learned_addr learned my addr 172.21.15.5:0/4276894903 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540e59c640 10 monclient(hunting): get_auth_request con 0x7f541006fe40 auth_method 0 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:10.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 10 monclient(hunting): get_auth_request con 0x7f54100a3e60 auth_method 0 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540ed9d640 1 --2- 172.21.15.5:0/4276894903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54100a5020 0x7f541013c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 10 monclient(hunting): handle_auth_done global_id 4218 payload 824 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 1 -- 172.21.15.5:0/4276894903 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541006fe40 msgr2=0x7f54101cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:10.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 1 --2- 172.21.15.5:0/4276894903 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541006fe40 0x7f54101cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 1 -- 172.21.15.5:0/4276894903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54100a5020 msgr2=0x7f541013c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 1 --2- 172.21.15.5:0/4276894903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54100a5020 0x7f541013c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 10 monclient: _finish_hunting 0 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 1 monclient: found mon.noname-a 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 1 -- 172.21.15.5:0/4276894903 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54100a12d0 con 0x7f54100a3e60 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540ed9d640 10 monclient: get_auth_request con 0x7f54100a5020 auth_method 0 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540ed9d640 1 --2- 172.21.15.5:0/4276894903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54100a5020 0x7f541013c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f540dd9b640 1 --2- 172.21.15.5:0/4276894903 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54100a3e60 0x7f54101d01d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f540800cea0 tx=0x7f540800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 1 -- 172.21.15.5:0/4276894903 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540801a070 con 0x7f54100a3e60 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 10 monclient: dump: 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 10 monclient: _finish_auth 0 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 1 -- 172.21.15.5:0/4276894903 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f540800d840 con 0x7f54100a3e60 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 1 -- 172.21.15.5:0/4276894903 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540800dc10 con 0x7f54100a3e60 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5406ffd640 10 monclient: dump: 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:10.190 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 5 monclient: authenticate success, global_id 4218 2024-09-13T11:40:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient: _renew_subs 2024-09-13T11:40:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 -- 172.21.15.5:0/4276894903 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541013fcc0 con 0x7f54100a3e60 2024-09-13T11:40:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5407fff640 4 set_mon_vals no callback set 2024-09-13T11:40:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient: _renew_subs 2024-09-13T11:40:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.171+0000 7f5414cb8640 1 -- 172.21.15.5:0/4276894903 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5410121220 con 0x7f54100a3e60 2024-09-13T11:40:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.175+0000 7f5414cb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.175+0000 7f5414cb8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:10.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.175+0000 7f5414cb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:10.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.175+0000 7f5406ffd640 1 -- 172.21.15.5:0/4276894903 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f540801d070 con 0x7f54100a3e60 2024-09-13T11:40:10.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.175+0000 7f5414cb8640 1 -- 172.21.15.5:0/4276894903 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54100a3920 con 0x7f54100a3e60 2024-09-13T11:40:10.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.175+0000 7f5406ffd640 1 --2- 172.21.15.5:0/4276894903 >> v2:172.21.15.39:6812/1340838781 conn(0x7f53f403b2d0 0x7f53f403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:10.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.175+0000 7f5406ffd640 1 -- 172.21.15.5:0/4276894903 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (secure 0 0 0) 0x7f540804f390 con 0x7f54100a3e60 2024-09-13T11:40:10.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.175+0000 7f540e59c640 1 --2- 172.21.15.5:0/4276894903 >> v2:172.21.15.39:6812/1340838781 conn(0x7f53f403b2d0 0x7f53f403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:10.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.175+0000 7f540e59c640 10 monclient: get_auth_request con 0x7f53f403b2d0 auth_method 0 2024-09-13T11:40:10.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.175+0000 7f540e59c640 1 --2- 172.21.15.5:0/4276894903 >> v2:172.21.15.39:6812/1340838781 conn(0x7f53f403b2d0 0x7f53f403d790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f53f801aa10 tx=0x7f53f80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:10.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.179+0000 7f5406ffd640 1 -- 172.21.15.5:0/4276894903 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f54100a12d0 con 0x7f54100a3e60 2024-09-13T11:40:10.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.179+0000 7f5406ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:10.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.179+0000 7f5406ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:10.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.351+0000 7f5414cb8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-13T11:40:10.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.351+0000 7f5414cb8640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-13T11:40:10.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.351+0000 7f5414cb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:10.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:10.351+0000 7f5414cb8640 1 -- 172.21.15.5:0/4276894903 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f541013f020 con 0x7f54100a3e60 2024-09-13T11:40:11.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.087+0000 7f5406ffd640 1 -- 172.21.15.5:0/4276894903 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (secure 0 0 0) 0x7f5408013340 con 0x7f54100a3e60 2024-09-13T11:40:11.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.087+0000 7f5406ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-13T11:40:11.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.087+0000 7f5406ffd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-13T11:40:11.090 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'rbd' created 2024-09-13T11:40:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.091+0000 7f5414cb8640 1 -- 172.21.15.5:0/4276894903 >> v2:172.21.15.39:6812/1340838781 conn(0x7f53f403b2d0 msgr2=0x7f53f403d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.091+0000 7f5414cb8640 1 --2- 172.21.15.5:0/4276894903 >> v2:172.21.15.39:6812/1340838781 conn(0x7f53f403b2d0 0x7f53f403d790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f53f801aa10 tx=0x7f53f80a6000 comp rx=0 tx=0).stop 2024-09-13T11:40:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.091+0000 7f5414cb8640 10 monclient: shutdown 2024-09-13T11:40:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.091+0000 7f5414cb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.091+0000 7f5414cb8640 1 -- 172.21.15.5:0/4276894903 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54100a3e60 msgr2=0x7f54101d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.091+0000 7f5414cb8640 1 --2- 172.21.15.5:0/4276894903 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54100a3e60 0x7f54101d01d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f540800cea0 tx=0x7f540800cf90 comp rx=0 tx=0).stop 2024-09-13T11:40:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.091+0000 7f5414cb8640 1 -- 172.21.15.5:0/4276894903 shutdown_connections 2024-09-13T11:40:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.091+0000 7f5414cb8640 1 --2- 172.21.15.5:0/4276894903 >> v2:172.21.15.39:6812/1340838781 conn(0x7f53f403b2d0 0x7f53f403d790 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f53f801aa10 tx=0x7f53f80a6000 comp rx=0 tx=0).stop 2024-09-13T11:40:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.091+0000 7f5414cb8640 1 --2- 172.21.15.5:0/4276894903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54100a5020 0x7f541013c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.091+0000 7f5414cb8640 1 --2- 172.21.15.5:0/4276894903 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54100a3e60 0x7f54101d01d0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f540800cea0 tx=0x7f540800cf90 comp rx=0 tx=0).stop 2024-09-13T11:40:11.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.091+0000 7f5414cb8640 1 --2- 172.21.15.5:0/4276894903 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f541006fe40 0x7f54101cfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:11.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.091+0000 7f5414cb8640 1 -- 172.21.15.5:0/4276894903 >> 172.21.15.5:0/4276894903 conn(0x7f54100a6be0 msgr2=0x7f54100a7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:11.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.095+0000 7f5414cb8640 1 -- 172.21.15.5:0/4276894903 shutdown_connections 2024-09-13T11:40:11.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.095+0000 7f5414cb8640 1 -- 172.21.15.5:0/4276894903 wait complete. 2024-09-13T11:40:11.121 DEBUG:teuthology.orchestra.run.smithi005:> rbd --cluster ceph pool init rbd 2024-09-13T11:40:11.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient: get_monmap_and_config 2024-09-13T11:40:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient: build_initial_monmap 2024-09-13T11:40:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient: monmap: 2024-09-13T11:40:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:11.186687+0000 2024-09-13T11:40:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:11.186687+0000 2024-09-13T11:40:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:11.186 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:11.186 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:11.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 1 Processor -- start 2024-09-13T11:40:11.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 1 -- start start 2024-09-13T11:40:11.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient: init 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient: _reopen_session rank -1 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 1 --2- >> v2:172.21.15.5:3300/0 conn(0x563dcec391d0 0x563dcec395b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient(hunting): picked mon.noname-a con 0x563dcec391d0 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 1 --2- >> v2:172.21.15.39:3300/0 conn(0x563dcec2ef10 0x563dcec39af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient(hunting): picked mon.noname-c con 0x563dcec2ef10 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient(hunting): picked mon.noname-c-legacy con 0x563dceccb020 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x563dcea85c50 con 0x563dceccb020 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x563dceb05a50 con 0x563dcec391d0 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x563dceb164c0 con 0x563dcec2ef10 2024-09-13T11:40:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient(hunting): _renew_subs 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.183+0000 7f612afd2100 10 monclient(hunting): authenticate will time out at 1014.535971s 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x563dcec391d0 0x563dcec395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x563dcec391d0 0x563dcec395b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57106/0 (socket says 172.21.15.5:57106) 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 1 -- 172.21.15.5:0/603963907 learned_addr learned my addr 172.21.15.5:0/603963907 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 10 monclient(hunting): get_auth_request con 0x563dcec391d0 auth_method 0 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f61292da640 1 --2- 172.21.15.5:0/603963907 >> v2:172.21.15.39:3300/0 conn(0x563dcec2ef10 0x563dcec39af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f61292da640 10 monclient(hunting): get_auth_request con 0x563dcec2ef10 auth_method 0 2024-09-13T11:40:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f61292da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f61292da640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f61292da640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 10 monclient(hunting): handle_auth_done global_id 4221 payload 274 2024-09-13T11:40:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 1 -- 172.21.15.5:0/603963907 >> v2:172.21.15.39:3300/0 conn(0x563dcec2ef10 msgr2=0x563dcec39af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 1 --2- 172.21.15.5:0/603963907 >> v2:172.21.15.39:3300/0 conn(0x563dcec2ef10 0x563dcec39af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 1 -- 172.21.15.5:0/603963907 >> v1:172.21.15.39:6789/0 conn(0x563dceccb020 legacy=0x563dceccb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 10 monclient: _finish_hunting 0 2024-09-13T11:40:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 1 monclient: found mon.noname-a 2024-09-13T11:40:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 1 -- 172.21.15.5:0/603963907 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dcec30150 con 0x563dcec391d0 2024-09-13T11:40:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.187+0000 7f6129adb640 1 --2- 172.21.15.5:0/603963907 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563dcec391d0 0x563dcec395b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f612000a820 tx=0x7f612001dbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db25d293b3b40c10 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 1 -- 172.21.15.5:0/603963907 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61200b3070 con 0x563dcec391d0 2024-09-13T11:40:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 10 monclient: dump: 2024-09-13T11:40:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 10 monclient: _finish_auth 0 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 10 monclient: _check_auth_tickets 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 1 -- 172.21.15.5:0/603963907 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61200a69b0 con 0x563dcec391d0 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 1 -- 172.21.15.5:0/603963907 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61200a6d80 con 0x563dcec391d0 2024-09-13T11:40:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 5 monclient: authenticate success, global_id 4221 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient: get_monmap_and_config success 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 4 set_mon_vals no callback set 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6128ad9640 10 monclient: dump: 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient: shutdown 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 -- 172.21.15.5:0/603963907 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563dcec391d0 msgr2=0x563dcec395b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 --2- 172.21.15.5:0/603963907 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563dcec391d0 0x563dcec395b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f612000a820 tx=0x7f612001dbd0 comp rx=0 tx=0).stop 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 -- 172.21.15.5:0/603963907 shutdown_connections 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 --2- 172.21.15.5:0/603963907 >> v2:172.21.15.39:3300/0 conn(0x563dcec2ef10 0x563dcec39af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 --2- 172.21.15.5:0/603963907 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563dcec391d0 0x563dcec395b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 -- 172.21.15.5:0/603963907 >> 172.21.15.5:0/603963907 conn(0x563dcec1b980 msgr2=0x563dcec1bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 -- 172.21.15.5:0/603963907 shutdown_connections 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 -- 172.21.15.5:0/603963907 wait complete. 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient: build_initial_monmap 2024-09-13T11:40:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient: monmap: 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 Processor -- start 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 -- start start 2024-09-13T11:40:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient: init 2024-09-13T11:40:11.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient: _reopen_session rank -1 2024-09-13T11:40:11.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:11.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563dcec2ef10 0x563dcec315e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient(hunting): picked mon.noname-c con 0x563dcec2ef10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563dcec391d0 0x563dcec31b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient(hunting): picked mon.noname-b con 0x563dcec391d0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563dceccb020 0x563dcec32060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient(hunting): picked mon.noname-a con 0x563dceccb020 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x563dceaaca40 con 0x563dceccb020 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563dceb09a00 con 0x563dcec391d0 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x563dcec30570 con 0x563dcec2ef10 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient(hunting): _renew_subs 2024-09-13T11:40:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612afd2100 10 monclient(hunting): authenticate will time out at 1014.543369s 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6129adb640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563dcec2ef10 0x563dcec315e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612ad3d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563dceccb020 0x563dcec32060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6129adb640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563dcec2ef10 0x563dcec315e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59036/0 (socket says 172.21.15.5:59036) 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612ad3d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563dceccb020 0x563dcec32060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57108/0 (socket says 172.21.15.5:57108) 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6129adb640 1 -- 172.21.15.5:0/932319635 learned_addr learned my addr 172.21.15.5:0/932319635 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6129adb640 10 monclient(hunting): get_auth_request con 0x563dcec2ef10 auth_method 0 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6129adb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6129adb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6129adb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612ad3d640 10 monclient(hunting): get_auth_request con 0x563dceccb020 auth_method 0 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612ad3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612ad3d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612ad3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6129adb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6129adb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6129adb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612ad3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612ad3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f612ad3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f61292da640 1 --2- 172.21.15.5:0/932319635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563dcec391d0 0x563dcec31b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f61292da640 10 monclient(hunting): get_auth_request con 0x563dcec391d0 auth_method 0 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f61292da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f61292da640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f61292da640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.191+0000 7f6129adb640 10 monclient(hunting): handle_auth_done global_id 4205 payload 824 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6129adb640 1 -- 172.21.15.5:0/932319635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563dcec391d0 msgr2=0x563dcec31b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6129adb640 1 --2- 172.21.15.5:0/932319635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563dcec391d0 0x563dcec31b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6129adb640 1 -- 172.21.15.5:0/932319635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563dceccb020 msgr2=0x563dcec32060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6129adb640 1 --2- 172.21.15.5:0/932319635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563dceccb020 0x563dcec32060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6129adb640 10 monclient: _finish_hunting 0 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6129adb640 1 monclient: found mon.noname-c 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6129adb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6129adb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6129adb640 1 -- 172.21.15.5:0/932319635 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dcec34e40 con 0x563dcec2ef10 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f61292da640 1 --2- 172.21.15.5:0/932319635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563dcec391d0 0x563dcec31b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f612ad3d640 1 --2- 172.21.15.5:0/932319635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x563dceccb020 0x563dcec32060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6129adb640 1 --2- 172.21.15.5:0/932319635 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563dcec2ef10 0x563dcec315e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f61200b1040 tx=0x7f612001a970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61200b3070 con 0x563dcec2ef10 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 10 monclient: dump: 2024-09-13T11:40:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:11.206 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61200bd070 con 0x563dcec2ef10 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61200aeb10 con 0x563dcec2ef10 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 10 monclient: dump: 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f611bfff640 4 set_mon_vals no callback set 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f612afd2100 5 monclient: authenticate success, global_id 4205 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f612afd2100 10 monclient: _renew_subs 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f612afd2100 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x563dcec35400 con 0x563dcec2ef10 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f612afd2100 10 monclient: _renew_subs 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f612afd2100 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x563dcec35120 con 0x563dcec2ef10 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f61200aecb0 con 0x563dcec2ef10 2024-09-13T11:40:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 1 --2- 172.21.15.5:0/932319635 >> v2:172.21.15.39:6812/1340838781 conn(0x7f6108039b90 0x7f610803c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3495+0+0 (secure 0 0 0) 0x7f61200eac20 con 0x563dcec2ef10 2024-09-13T11:40:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f612afd2100 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-13T11:40:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f612afd2100 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-13T11:40:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f612afd2100 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x563dcec337f0 con 0x563dcec2ef10 2024-09-13T11:40:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f61292da640 1 --2- 172.21.15.5:0/932319635 >> v2:172.21.15.39:6812/1340838781 conn(0x7f6108039b90 0x7f610803c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f61292da640 10 monclient: get_auth_request con 0x7f6108039b90 auth_method 0 2024-09-13T11:40:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:11.195+0000 7f61292da640 1 --2- 172.21.15.5:0/932319635 >> v2:172.21.15.39:6812/1340838781 conn(0x7f6108039b90 0x7f610803c050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6114005fd0 tx=0x7f6114005d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:12.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.099+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 141+0+0 (secure 0 0 0) 0x7f61200178e0 con 0x563dcec2ef10 2024-09-13T11:40:12.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.099+0000 7f6119ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-13T11:40:12.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.099+0000 7f6119ffb640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-13T11:40:12.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.099+0000 7f612afd2100 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:12.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.099+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f61200a69b0 con 0x563dcec2ef10 2024-09-13T11:40:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.107+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f61200d1050 con 0x563dcec2ef10 2024-09-13T11:40:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.107+0000 7f6119ffb640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-13T11:40:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.107+0000 7f611bfff640 10 monclient: _renew_subs 2024-09-13T11:40:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.107+0000 7f611bfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.107+0000 7f611bfff640 1 -- 172.21.15.5:0/932319635 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=17}) -- 0x7f610c002af0 con 0x563dcec2ef10 2024-09-13T11:40:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.107+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 <== mon.2 v2:172.21.15.5:3301/0 8 ==== osd_map(17..17 src has 1..17) ==== 658+0+0 (secure 0 0 0) 0x7f61200c0d70 con 0x563dcec2ef10 2024-09-13T11:40:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.107+0000 7f612afd2100 1 --2- 172.21.15.5:0/932319635 >> v2:172.21.15.39:6808/1185247880 conn(0x563dcecc3bb0 0x563dcece57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.107+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 --> v2:172.21.15.39:6808/1185247880 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x563dcece5d20 con 0x563dcecc3bb0 2024-09-13T11:40:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.107+0000 7f612ad3d640 1 --2- 172.21.15.5:0/932319635 >> v2:172.21.15.39:6808/1185247880 conn(0x563dcecc3bb0 0x563dcece57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.107+0000 7f612ad3d640 10 monclient: get_auth_request con 0x563dcecc3bb0 auth_method 0 2024-09-13T11:40:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.111+0000 7f612ad3d640 1 --2- 172.21.15.5:0/932319635 >> v2:172.21.15.39:6808/1185247880 conn(0x563dcecc3bb0 0x563dcece57e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.115+0000 7f612ad3d640 1 -- 172.21.15.5:0/932319635 <== osd.5 v2:172.21.15.39:6808/1185247880 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f611c00d040 con 0x563dcecc3bb0 2024-09-13T11:40:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.119+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 --> v2:172.21.15.39:6808/1185247880 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x563dced38980 con 0x563dcecc3bb0 2024-09-13T11:40:12.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.119+0000 7f612ad3d640 1 -- 172.21.15.5:0/932319635 <== osd.5 v2:172.21.15.39:6808/1185247880 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f611c00d040 con 0x563dcecc3bb0 2024-09-13T11:40:12.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.119+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 --> v2:172.21.15.39:6808/1185247880 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x563dced37a90 con 0x563dcecc3bb0 2024-09-13T11:40:12.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.123+0000 7f612ad3d640 1 -- 172.21.15.5:0/932319635 <== osd.5 v2:172.21.15.39:6808/1185247880 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f611c00d040 con 0x563dcecc3bb0 2024-09-13T11:40:12.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.123+0000 7f611b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:12.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:12.123+0000 7f611b7fe640 1 -- 172.21.15.5:0/932319635 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f6100001f10 con 0x563dcec2ef10 2024-09-13T11:40:13.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:13.115+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 <== mon.2 v2:172.21.15.5:3301/0 9 ==== pool_op_reply(tid 4 (0) Success v18) ==== 55+0+0 (secure 0 0 0) 0x7f61200acc10 con 0x563dcec2ef10 2024-09-13T11:40:13.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:13.115+0000 7f6119ffb640 10 monclient: _renew_subs 2024-09-13T11:40:13.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:13.115+0000 7f6119ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:13.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:13.115+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=18}) -- 0x7f610803fab0 con 0x563dcec2ef10 2024-09-13T11:40:13.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:13.115+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 <== mon.2 v2:172.21.15.5:3301/0 10 ==== osd_map(18..18 src has 1..18) ==== 626+0+0 (secure 0 0 0) 0x7f61200dc030 con 0x563dcec2ef10 2024-09-13T11:40:13.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:13.119+0000 7f611bfff640 1 -- 172.21.15.5:0/932319635 --> v2:172.21.15.39:6808/1185247880 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7f610c0057c0 con 0x563dcecc3bb0 2024-09-13T11:40:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:13.123+0000 7f612ad3d640 1 -- 172.21.15.5:0/932319635 <== osd.5 v2:172.21.15.39:6808/1185247880 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f611c00d040 con 0x563dcecc3bb0 2024-09-13T11:40:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:13.123+0000 7f611b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:13.123+0000 7f611b7fe640 1 -- 172.21.15.5:0/932319635 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) -- 0x7f6100003360 con 0x563dcec2ef10 2024-09-13T11:40:14.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.127+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 <== mon.2 v2:172.21.15.5:3301/0 11 ==== pool_op_reply(tid 6 (0) Success v19) ==== 43+0+0 (secure 0 0 0) 0x7f61200acdf0 con 0x563dcec2ef10 2024-09-13T11:40:14.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.127+0000 7f6119ffb640 10 monclient: _renew_subs 2024-09-13T11:40:14.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.127+0000 7f6119ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:14.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.127+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=19}) -- 0x7f610803fb60 con 0x563dcec2ef10 2024-09-13T11:40:14.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.127+0000 7f6119ffb640 1 -- 172.21.15.5:0/932319635 <== mon.2 v2:172.21.15.5:3301/0 12 ==== osd_map(19..19 src has 1..19) ==== 654+0+0 (secure 0 0 0) 0x7f61200e5070 con 0x563dcec2ef10 2024-09-13T11:40:14.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.127+0000 7f611bfff640 1 -- 172.21.15.5:0/932319635 --> v2:172.21.15.39:6808/1185247880 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7f610c005b60 con 0x563dcecc3bb0 2024-09-13T11:40:14.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612ad3d640 1 -- 172.21.15.5:0/932319635 <== osd.5 v2:172.21.15.39:6808/1185247880 5 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f611c00d040 con 0x563dcecc3bb0 2024-09-13T11:40:14.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 >> v2:172.21.15.39:6808/1185247880 conn(0x563dcecc3bb0 msgr2=0x563dcece57e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612afd2100 1 --2- 172.21.15.5:0/932319635 >> v2:172.21.15.39:6808/1185247880 conn(0x563dcecc3bb0 0x563dcece57e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 >> v2:172.21.15.39:6812/1340838781 conn(0x7f6108039b90 msgr2=0x7f610803c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612afd2100 1 --2- 172.21.15.5:0/932319635 >> v2:172.21.15.39:6812/1340838781 conn(0x7f6108039b90 0x7f610803c050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6114005fd0 tx=0x7f6114005d60 comp rx=0 tx=0).stop 2024-09-13T11:40:14.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612afd2100 10 monclient: shutdown 2024-09-13T11:40:14.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612afd2100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:14.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563dcec2ef10 msgr2=0x563dcec315e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612afd2100 1 --2- 172.21.15.5:0/932319635 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x563dcec2ef10 0x563dcec315e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f61200b1040 tx=0x7f612001a970 comp rx=0 tx=0).stop 2024-09-13T11:40:14.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612ad3d640 1 -- 172.21.15.5:0/932319635 reap_dead start 2024-09-13T11:40:14.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 shutdown_connections 2024-09-13T11:40:14.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 >> 172.21.15.5:0/932319635 conn(0x563dcec1b980 msgr2=0x563dcec3b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:14.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 shutdown_connections 2024-09-13T11:40:14.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.139+0000 7f612afd2100 1 -- 172.21.15.5:0/932319635 wait complete. 2024-09-13T11:40:14.149 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T11:40:14.150 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-13T11:40:14.150 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T11:40:14.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient: get_monmap_and_config 2024-09-13T11:40:14.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient: build_initial_monmap 2024-09-13T11:40:14.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient: monmap: 2024-09-13T11:40:14.284 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:14.284 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.284 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:14.285970+0000 2024-09-13T11:40:14.284 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:14.285970+0000 2024-09-13T11:40:14.284 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:14.284 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.285 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:14.285 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:14.285 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:14.285 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:14.285 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:14.285 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 1 Processor -- start 2024-09-13T11:40:14.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 1 -- start start 2024-09-13T11:40:14.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient: init 2024-09-13T11:40:14.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:14.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-13T11:40:14.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5a88137930 0x7f5a8809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient(hunting): picked mon.noname-a con 0x7f5a88137930 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:14.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a88137d10 0x7f5a881380f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient(hunting): picked mon.noname-b con 0x7f5a88137d10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:14.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5a880a4a70 0x7f5a880a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient(hunting): picked mon.noname-c con 0x7f5a880a4a70 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f5a880a5fe0 con 0x7f5a88137930 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f5a8809c540 con 0x7f5a880a4a70 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f5a8809bef0 con 0x7f5a88137d10 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8f9a7640 10 monclient(hunting): authenticate will time out at 1017.634760s 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8cf42640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a88137d10 0x7f5a881380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5a88137930 0x7f5a8809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5a88137930 0x7f5a8809cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57114/0 (socket says 172.21.15.5:57114) 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8cf42640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a88137d10 0x7f5a881380f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59042/0 (socket says 172.21.15.5:59042) 2024-09-13T11:40:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 1 -- 172.21.15.5:0/1333732777 learned_addr learned my addr 172.21.15.5:0/1333732777 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 10 monclient(hunting): get_auth_request con 0x7f5a88137930 auth_method 0 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8cf42640 10 monclient(hunting): get_auth_request con 0x7f5a88137d10 auth_method 0 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8cf42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8cf42640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8cf42640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8cf42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8cf42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8cf42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8df44640 1 --2- 172.21.15.5:0/1333732777 >> v2:172.21.15.39:3300/0 conn(0x7f5a880a4a70 0x7f5a880a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8df44640 10 monclient(hunting): get_auth_request con 0x7f5a880a4a70 auth_method 0 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8df44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8df44640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8df44640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 1 -- 172.21.15.5:0/1333732777 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a88137d10 msgr2=0x7f5a881380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 1 --2- 172.21.15.5:0/1333732777 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a88137d10 0x7f5a881380f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 1 -- 172.21.15.5:0/1333732777 >> v2:172.21.15.39:3300/0 conn(0x7f5a880a4a70 msgr2=0x7f5a880a4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 1 --2- 172.21.15.5:0/1333732777 >> v2:172.21.15.39:3300/0 conn(0x7f5a880a4a70 0x7f5a880a4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 10 monclient: _finish_hunting 0 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 1 monclient: found mon.noname-a 2024-09-13T11:40:14.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:14.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:14.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8d743640 1 -- 172.21.15.5:0/1333732777 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a8809c1d0 con 0x7f5a88137930 2024-09-13T11:40:14.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.283+0000 7f5a8cf42640 1 --2- 172.21.15.5:0/1333732777 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a88137d10 0x7f5a881380f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:14.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8d743640 1 --2- 172.21.15.5:0/1333732777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a88137930 0x7f5a8809cc90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009870 tx=0x7f5a7c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792b0df40e986e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:14.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 1 -- 172.21.15.5:0/1333732777 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c0b0070 con 0x7f5a88137930 2024-09-13T11:40:14.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:14.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:14.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 10 monclient: dump: 2024-09-13T11:40:14.298 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:14.298 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.298 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.298 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.298 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:14.298 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.298 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 10 monclient: _finish_auth 0 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 1 -- 172.21.15.5:0/1333732777 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a7c0ab600 con 0x7f5a88137930 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 1 -- 172.21.15.5:0/1333732777 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c0ab9d0 con 0x7f5a88137930 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 5 monclient: authenticate success, global_id 4227 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 4 set_mon_vals no callback set 2024-09-13T11:40:14.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a7bfff640 10 monclient: dump: 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 10 monclient: shutdown 2024-09-13T11:40:14.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/1333732777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a88137930 msgr2=0x7f5a8809cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 1 --2- 172.21.15.5:0/1333732777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a88137930 0x7f5a8809cc90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009870 tx=0x7f5a7c01c380 comp rx=0 tx=0).stop 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/1333732777 shutdown_connections 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 1 --2- 172.21.15.5:0/1333732777 >> v2:172.21.15.39:3300/0 conn(0x7f5a880a4a70 0x7f5a880a4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 1 --2- 172.21.15.5:0/1333732777 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a88137d10 0x7f5a881380f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 1 --2- 172.21.15.5:0/1333732777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a88137930 0x7f5a8809cc90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/1333732777 >> 172.21.15.5:0/1333732777 conn(0x7f5a880a6c70 msgr2=0x7f5a880a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/1333732777 shutdown_connections 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/1333732777 wait complete. 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 10 monclient: build_initial_monmap 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.287+0000 7f5a8f9a7640 10 monclient: monmap: 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:14.301 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 1 Processor -- start 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 1 -- start start 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient: init 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a880a4a70 0x7f5a881cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient(hunting): picked mon.noname-c con 0x7f5a880a4a70 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a88137930 0x7f5a88136a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient(hunting): picked mon.noname-b con 0x7f5a88137930 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a88137d10 0x7f5a8813cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient(hunting): picked mon.noname-a con 0x7f5a88137d10 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f5a88141190 con 0x7f5a88137d10 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5a88141010 con 0x7f5a88137930 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f5a88141310 con 0x7f5a880a4a70 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:14.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient(hunting): authenticate will time out at 1017.640972s 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a880a4a70 0x7f5a881cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a880a4a70 0x7f5a881cf6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59052/0 (socket says 172.21.15.5:59052) 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 1 -- 172.21.15.5:0/806386255 learned_addr learned my addr 172.21.15.5:0/806386255 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 10 monclient(hunting): get_auth_request con 0x7f5a880a4a70 auth_method 0 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8df44640 1 --2- 172.21.15.5:0/806386255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a88137d10 0x7f5a8813cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8df44640 10 monclient(hunting): get_auth_request con 0x7f5a88137d10 auth_method 0 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8df44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8df44640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8df44640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8df44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8df44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8df44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8cf42640 1 --2- 172.21.15.5:0/806386255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a88137930 0x7f5a88136a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 10 monclient(hunting): handle_auth_done global_id 4211 payload 824 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 1 -- 172.21.15.5:0/806386255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a88137930 msgr2=0x7f5a88136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 1 --2- 172.21.15.5:0/806386255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a88137930 0x7f5a88136a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 1 -- 172.21.15.5:0/806386255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a88137d10 msgr2=0x7f5a8813cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 1 --2- 172.21.15.5:0/806386255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a88137d10 0x7f5a8813cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 10 monclient: _finish_hunting 0 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 1 monclient: found mon.noname-c 2024-09-13T11:40:14.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 1 -- 172.21.15.5:0/806386255 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a88122400 con 0x7f5a880a4a70 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8cf42640 10 monclient: get_auth_request con 0x7f5a88137930 auth_method 0 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8cf42640 1 --2- 172.21.15.5:0/806386255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a88137930 0x7f5a88136a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8d743640 1 --2- 172.21.15.5:0/806386255 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a880a4a70 0x7f5a881cf6f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0ae040 tx=0x7f5a7c019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 1 -- 172.21.15.5:0/806386255 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c0b0040 con 0x7f5a880a4a70 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 10 monclient: dump: 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.306 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 1 -- 172.21.15.5:0/806386255 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a7c0b4420 con 0x7f5a880a4a70 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 1 -- 172.21.15.5:0/806386255 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c0b4810 con 0x7f5a880a4a70 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 10 monclient: dump: 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a7b7fe640 4 set_mon_vals no callback set 2024-09-13T11:40:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 5 monclient: authenticate success, global_id 4211 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient: _renew_subs 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/806386255 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a880a0e90 con 0x7f5a880a4a70 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient: _renew_subs 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/806386255 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a88122120 con 0x7f5a880a4a70 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 1 -- 172.21.15.5:0/806386255 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f5a7c0b49b0 con 0x7f5a880a4a70 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/806386255 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a8809cc90 con 0x7f5a880a4a70 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 1 --2- 172.21.15.5:0/806386255 >> v2:172.21.15.39:6812/1340838781 conn(0x7f5a6c03b2a0 0x7f5a6c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.291+0000 7f5a79ffb640 1 -- 172.21.15.5:0/806386255 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f5a7c0e89e0 con 0x7f5a880a4a70 2024-09-13T11:40:14.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.295+0000 7f5a8cf42640 1 --2- 172.21.15.5:0/806386255 >> v2:172.21.15.39:6812/1340838781 conn(0x7f5a6c03b2a0 0x7f5a6c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.295+0000 7f5a8cf42640 10 monclient: get_auth_request con 0x7f5a6c03b2a0 auth_method 0 2024-09-13T11:40:14.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.295+0000 7f5a8cf42640 1 --2- 172.21.15.5:0/806386255 >> v2:172.21.15.39:6812/1340838781 conn(0x7f5a6c03b2a0 0x7f5a6c03d760 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5a8813ecb0 tx=0x7f5a84008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:14.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.299+0000 7f5a79ffb640 1 -- 172.21.15.5:0/806386255 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f5a7c0160f0 con 0x7f5a880a4a70 2024-09-13T11:40:14.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.299+0000 7f5a79ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:14.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.299+0000 7f5a79ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:14.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.471+0000 7f5a8f9a7640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-13T11:40:14.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.471+0000 7f5a8f9a7640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-13T11:40:14.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.471+0000 7f5a8f9a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:14.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.471+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/806386255 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f5a881d0570 con 0x7f5a880a4a70 2024-09-13T11:40:14.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.471+0000 7f5a79ffb640 1 -- 172.21.15.5:0/806386255 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v2) ==== 86+0+159 (secure 0 0 0) 0x7f5a7c0bc360 con 0x7f5a880a4a70 2024-09-13T11:40:14.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.471+0000 7f5a79ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-13T11:40:14.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.471+0000 7f5a79ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:40:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/806386255 >> v2:172.21.15.39:6812/1340838781 conn(0x7f5a6c03b2a0 msgr2=0x7f5a6c03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 1 --2- 172.21.15.5:0/806386255 >> v2:172.21.15.39:6812/1340838781 conn(0x7f5a6c03b2a0 0x7f5a6c03d760 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5a8813ecb0 tx=0x7f5a84008040 comp rx=0 tx=0).stop 2024-09-13T11:40:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 10 monclient: shutdown 2024-09-13T11:40:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/806386255 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a880a4a70 msgr2=0x7f5a881cf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 1 --2- 172.21.15.5:0/806386255 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a880a4a70 0x7f5a881cf6f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0ae040 tx=0x7f5a7c019180 comp rx=0 tx=0).stop 2024-09-13T11:40:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/806386255 shutdown_connections 2024-09-13T11:40:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 1 --2- 172.21.15.5:0/806386255 >> v2:172.21.15.39:6812/1340838781 conn(0x7f5a6c03b2a0 0x7f5a6c03d760 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 1 --2- 172.21.15.5:0/806386255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a88137d10 0x7f5a8813cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 1 --2- 172.21.15.5:0/806386255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a88137930 0x7f5a88136a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 1 --2- 172.21.15.5:0/806386255 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5a880a4a70 0x7f5a881cf6f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/806386255 >> 172.21.15.5:0/806386255 conn(0x7f5a880a6c70 msgr2=0x7f5a8809fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:14.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/806386255 shutdown_connections 2024-09-13T11:40:14.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.475+0000 7f5a8f9a7640 1 -- 172.21.15.5:0/806386255 wait complete. 2024-09-13T11:40:14.492 INFO:teuthology.orchestra.run.smithi005.stdout:[{"version":2,"timestamp":"2024-09-13T11:40:02.052522+0000","name":"","changes":[{"name":"osd.2/osd_mclock_max_capacity_iops_ssd","new_value":"7555.261030"}]}] 2024-09-13T11:40:14.492 INFO:tasks.ceph_manager:config epoch is 2 2024-09-13T11:40:14.492 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T11:40:14.493 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T11:40:14.493 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T11:40:14.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient: get_monmap_and_config 2024-09-13T11:40:14.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient: build_initial_monmap 2024-09-13T11:40:14.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient: monmap: 2024-09-13T11:40:14.611 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:14.611 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.611 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:14.612927+0000 2024-09-13T11:40:14.611 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:14.612927+0000 2024-09-13T11:40:14.611 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:14.611 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.612 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:14.612 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:14.612 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:14.612 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:14.612 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:14.612 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 1 Processor -- start 2024-09-13T11:40:14.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 1 -- start start 2024-09-13T11:40:14.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient: init 2024-09-13T11:40:14.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:14.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-13T11:40:14.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2e94137930 0x7f2e9409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient(hunting): picked mon.noname-a con 0x7f2e94137930 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e94137d10 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e9413cd80 0x7f2e9413f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient(hunting): picked mon.noname-b con 0x7f2e9413cd80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e940a5a80 con 0x7f2e94137d10 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f2e940a5fe0 con 0x7f2e94137930 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f2e9409c540 con 0x7f2e9413cd80 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9c124640 10 monclient(hunting): authenticate will time out at 1017.961670s 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9a6c1640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e9413cd80 0x7f2e9413f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2e94137930 0x7f2e9409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9a6c1640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e9413cd80 0x7f2e9413f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59058/0 (socket says 172.21.15.5:59058) 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 10 monclient(hunting): get_auth_request con 0x7f2e94137930 auth_method 0 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9a6c1640 1 -- 172.21.15.5:0/940497727 learned_addr learned my addr 172.21.15.5:0/940497727 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9a6c1640 10 monclient(hunting): get_auth_request con 0x7f2e9413cd80 auth_method 0 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9a6c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9a6c1640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9a6c1640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:14.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9a6c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9a6c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9a6c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 10 monclient(hunting): handle_auth_done global_id 4233 payload 274 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 1 -- 172.21.15.5:0/940497727 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e9413cd80 msgr2=0x7f2e9413f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 1 --2- 172.21.15.5:0/940497727 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e9413cd80 0x7f2e9413f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 1 -- 172.21.15.5:0/940497727 >> v1:172.21.15.5:6789/0 conn(0x7f2e94137d10 legacy=0x7f2e941380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 10 monclient: _finish_hunting 0 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 1 monclient: found mon.noname-a 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e99ec0640 1 -- 172.21.15.5:0/940497727 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9409bef0 con 0x7f2e94137930 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e98ebe640 1 -- 172.21.15.5:0/940497727 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1735956808 0 0) 0x7f2e940a5a80 con 0x7f2e94137d10 2024-09-13T11:40:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e98ebe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:40:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.611+0000 7f2e9a6c1640 1 --2- 172.21.15.5:0/940497727 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e9413cd80 0x7f2e9413f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 1 --2- 172.21.15.5:0/940497727 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e94137930 0x7f2e9409cc90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c0097c0 tx=0x7f2e7c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c06078aefadc87 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 1 -- 172.21.15.5:0/940497727 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c0b0070 con 0x7f2e94137930 2024-09-13T11:40:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 10 monclient: dump: 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 10 monclient: _finish_auth 0 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 5 monclient: authenticate success, global_id 4233 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 1 -- 172.21.15.5:0/940497727 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e7c0ab600 con 0x7f2e94137930 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 4 set_mon_vals no callback set 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 1 -- 172.21.15.5:0/940497727 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c0ab9d0 con 0x7f2e94137930 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e98ebe640 10 monclient: dump: 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient: shutdown 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 -- 172.21.15.5:0/940497727 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e94137930 msgr2=0x7f2e9409cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 --2- 172.21.15.5:0/940497727 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e94137930 0x7f2e9409cc90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c0097c0 tx=0x7f2e7c01c360 comp rx=0 tx=0).stop 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 -- 172.21.15.5:0/940497727 shutdown_connections 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 --2- 172.21.15.5:0/940497727 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e9413cd80 0x7f2e9413f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 --2- 172.21.15.5:0/940497727 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e94137930 0x7f2e9409cc90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 -- 172.21.15.5:0/940497727 >> 172.21.15.5:0/940497727 conn(0x7f2e940a6c70 msgr2=0x7f2e940a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 -- 172.21.15.5:0/940497727 shutdown_connections 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 -- 172.21.15.5:0/940497727 wait complete. 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient: build_initial_monmap 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient: monmap: 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 Processor -- start 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 -- start start 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient: init 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e94137930 0x7f2e94073ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient(hunting): picked mon.noname-a con 0x7f2e94137930 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e94137d10 0x7f2e94139ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient(hunting): picked mon.noname-c con 0x7f2e94137d10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e9413cd80 0x7f2e9413a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient(hunting): picked mon.noname-b con 0x7f2e9413cd80 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2e94142310 con 0x7f2e94137930 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2e94142190 con 0x7f2e9413cd80 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f2e94142490 con 0x7f2e94137d10 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient(hunting): authenticate will time out at 1017.966798s 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e94137930 0x7f2e94073ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e94137930 0x7f2e94073ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57154/0 (socket says 172.21.15.5:57154) 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 1 -- 172.21.15.5:0/548692367 learned_addr learned my addr 172.21.15.5:0/548692367 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 10 monclient(hunting): get_auth_request con 0x7f2e94137930 auth_method 0 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e996bf640 1 --2- 172.21.15.5:0/548692367 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e94137d10 0x7f2e94139ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e996bf640 10 monclient(hunting): get_auth_request con 0x7f2e94137d10 auth_method 0 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e996bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e996bf640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e996bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e996bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e996bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e996bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-09-13T11:40:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 1 -- 172.21.15.5:0/548692367 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e94137d10 msgr2=0x7f2e94139ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 1 --2- 172.21.15.5:0/548692367 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e94137d10 0x7f2e94139ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 1 -- 172.21.15.5:0/548692367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e9413cd80 msgr2=0x7f2e9413a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9a6c1640 1 --2- 172.21.15.5:0/548692367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e9413cd80 0x7f2e9413a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 1 --2- 172.21.15.5:0/548692367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e9413cd80 0x7f2e9413a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 10 monclient: _finish_hunting 0 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 1 monclient: found mon.noname-a 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 1 -- 172.21.15.5:0/548692367 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e94071470 con 0x7f2e94137930 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e99ec0640 1 --2- 172.21.15.5:0/548692367 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e94137930 0x7f2e94073ea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c019f90 tx=0x7f2e7c019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 1 -- 172.21.15.5:0/548692367 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c0b8020 con 0x7f2e94137930 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:14.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 10 monclient: dump: 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 5 monclient: authenticate success, global_id 4239 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 1 -- 172.21.15.5:0/548692367 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e7c0b4410 con 0x7f2e94137930 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient: _renew_subs 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 -- 172.21.15.5:0/548692367 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e940a9080 con 0x7f2e94137930 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 1 -- 172.21.15.5:0/548692367 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c0b0040 con 0x7f2e94137930 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e927fc640 10 monclient: dump: 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e93fff640 4 set_mon_vals no callback set 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient: _renew_subs 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.615+0000 7f2e9c124640 1 -- 172.21.15.5:0/548692367 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e94070eb0 con 0x7f2e94137930 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.619+0000 7f2e9c124640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.619+0000 7f2e9c124640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.619+0000 7f2e9c124640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.619+0000 7f2e9c124640 1 -- 172.21.15.5:0/548692367 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e9409cc90 con 0x7f2e94137930 2024-09-13T11:40:14.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.619+0000 7f2e927fc640 1 -- 172.21.15.5:0/548692367 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f2e7c0bb020 con 0x7f2e94137930 2024-09-13T11:40:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.619+0000 7f2e927fc640 1 --2- 172.21.15.5:0/548692367 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2e7403b2a0 0x7f2e7403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.619+0000 7f2e927fc640 1 -- 172.21.15.5:0/548692367 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f2e7c0e8950 con 0x7f2e94137930 2024-09-13T11:40:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.619+0000 7f2e996bf640 1 --2- 172.21.15.5:0/548692367 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2e7403b2a0 0x7f2e7403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.619+0000 7f2e996bf640 10 monclient: get_auth_request con 0x7f2e7403b2a0 auth_method 0 2024-09-13T11:40:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.619+0000 7f2e996bf640 1 --2- 172.21.15.5:0/548692367 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2e7403b2a0 0x7f2e7403d760 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2e94137900 tx=0x7f2e84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.623+0000 7f2e927fc640 1 -- 172.21.15.5:0/548692367 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f2e7c0c2030 con 0x7f2e94137930 2024-09-13T11:40:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.623+0000 7f2e927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.623+0000 7f2e927fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:14.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.835+0000 7f2e9c124640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-13T11:40:14.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.835+0000 7f2e9c124640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-13T11:40:14.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.835+0000 7f2e9c124640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:14.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.835+0000 7f2e9c124640 1 -- 172.21.15.5:0/548692367 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f2e9406cbe0 con 0x7f2e94137930 2024-09-13T11:40:14.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.835+0000 7f2e927fc640 1 -- 172.21.15.5:0/548692367 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+92849 (secure 0 0 0) 0x7f2e7c016070 con 0x7f2e94137930 2024-09-13T11:40:14.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.835+0000 7f2e927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-13T11:40:14.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.835+0000 7f2e927fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:14.839 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:40:14.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 1 -- 172.21.15.5:0/548692367 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2e7403b2a0 msgr2=0x7f2e7403d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 1 --2- 172.21.15.5:0/548692367 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2e7403b2a0 0x7f2e7403d760 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2e94137900 tx=0x7f2e84009040 comp rx=0 tx=0).stop 2024-09-13T11:40:14.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 10 monclient: shutdown 2024-09-13T11:40:14.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:14.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 1 -- 172.21.15.5:0/548692367 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e94137930 msgr2=0x7f2e94073ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 1 --2- 172.21.15.5:0/548692367 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e94137930 0x7f2e94073ea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c019f90 tx=0x7f2e7c019100 comp rx=0 tx=0).stop 2024-09-13T11:40:14.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 1 -- 172.21.15.5:0/548692367 shutdown_connections 2024-09-13T11:40:14.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 1 --2- 172.21.15.5:0/548692367 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2e7403b2a0 0x7f2e7403d760 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 1 --2- 172.21.15.5:0/548692367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e9413cd80 0x7f2e9413a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 1 --2- 172.21.15.5:0/548692367 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2e94137d10 0x7f2e94139ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 1 --2- 172.21.15.5:0/548692367 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e94137930 0x7f2e94073ea0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 1 -- 172.21.15.5:0/548692367 >> 172.21.15.5:0/548692367 conn(0x7f2e940a6c70 msgr2=0x7f2e9413dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:14.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 1 -- 172.21.15.5:0/548692367 shutdown_connections 2024-09-13T11:40:14.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.839+0000 7f2e9c124640 1 -- 172.21.15.5:0/548692367 wait complete. 2024-09-13T11:40:14.855 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1340838781}]},"active_addr":"172.21.15.39:6812/1340838781","active_change":"2024-09-13T11:39:58.286053+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":3813290939}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":2677173742}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":1248720508}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":352016487}]}]} 2024-09-13T11:40:14.858 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T11:40:14.858 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T11:40:14.858 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:40:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient: get_monmap_and_config 2024-09-13T11:40:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient: build_initial_monmap 2024-09-13T11:40:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient: monmap: 2024-09-13T11:40:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:14.970141+0000 2024-09-13T11:40:14.969 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:14.970141+0000 2024-09-13T11:40:14.969 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:14.969 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.969 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:14.969 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:14.969 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:14.969 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:14.969 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:14.969 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 1 Processor -- start 2024-09-13T11:40:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 1 -- start start 2024-09-13T11:40:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient: init 2024-09-13T11:40:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-13T11:40:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb0900a0ea0 0x7fb0900a1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0900a0ea0 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb0900a1ae0 0x7fb0901354b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0900a1ae0 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0901359f0 0x7fb090135e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0901359f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fb09006f640 con 0x7fb0900a1ae0 2024-09-13T11:40:14.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fb0900a0000 con 0x7fb0900a0ea0 2024-09-13T11:40:14.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fb09006f010 con 0x7fb0901359f0 2024-09-13T11:40:14.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:14.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb094e5b640 10 monclient(hunting): authenticate will time out at 1018.318873s 2024-09-13T11:40:14.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08dd9b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb0900a1ae0 0x7fb0901354b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0901359f0 0x7fb090135e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0901359f0 0x7fb090135e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59076/0 (socket says 172.21.15.5:59076) 2024-09-13T11:40:14.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08dd9b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb0900a1ae0 0x7fb0901354b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57156/0 (socket says 172.21.15.5:57156) 2024-09-13T11:40:14.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 1 -- 172.21.15.5:0/1275244035 learned_addr learned my addr 172.21.15.5:0/1275244035 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 10 monclient(hunting): get_auth_request con 0x7fb0901359f0 auth_method 0 2024-09-13T11:40:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08dd9b640 10 monclient(hunting): get_auth_request con 0x7fb0900a1ae0 auth_method 0 2024-09-13T11:40:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08e59c640 1 --2- 172.21.15.5:0/1275244035 >> v2:172.21.15.39:3300/0 conn(0x7fb0900a0ea0 0x7fb0900a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2024-09-13T11:40:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 1 -- 172.21.15.5:0/1275244035 >> v2:172.21.15.39:3300/0 conn(0x7fb0900a0ea0 msgr2=0x7fb0900a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 1 --2- 172.21.15.5:0/1275244035 >> v2:172.21.15.39:3300/0 conn(0x7fb0900a0ea0 0x7fb0900a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 1 -- 172.21.15.5:0/1275244035 >> v2:172.21.15.5:3300/0 conn(0x7fb0900a1ae0 msgr2=0x7fb0901354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 1 --2- 172.21.15.5:0/1275244035 >> v2:172.21.15.5:3300/0 conn(0x7fb0900a1ae0 0x7fb0901354b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 10 monclient: _finish_hunting 0 2024-09-13T11:40:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 1 monclient: found mon.noname-b 2024-09-13T11:40:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:40:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08ed9d640 1 -- 172.21.15.5:0/1275244035 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09009fd80 con 0x7fb0901359f0 2024-09-13T11:40:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08e59c640 10 monclient: get_auth_request con 0x7fb0900a0ea0 auth_method 0 2024-09-13T11:40:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.967+0000 7fb08e59c640 1 --2- 172.21.15.5:0/1275244035 >> v2:172.21.15.39:3300/0 conn(0x7fb0900a0ea0 0x7fb0900a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08ed9d640 1 --2- 172.21.15.5:0/1275244035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0901359f0 0x7fb090135e40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb0900a0210 tx=0x7fb08001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb5bd390a5d2f3e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 1 -- 172.21.15.5:0/1275244035 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0800b0070 con 0x7fb0901359f0 2024-09-13T11:40:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 10 monclient: dump: 2024-09-13T11:40:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.979 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.979 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:14.979 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.979 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:14.979 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:14.979 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:14.979 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 10 monclient: _finish_auth 0 2024-09-13T11:40:14.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 10 monclient: _check_auth_tickets 2024-09-13T11:40:14.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:14.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 1 -- 172.21.15.5:0/1275244035 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0800a4e50 con 0x7fb0901359f0 2024-09-13T11:40:14.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:14.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 1 -- 172.21.15.5:0/1275244035 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0800ab820 con 0x7fb0901359f0 2024-09-13T11:40:14.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 5 monclient: authenticate success, global_id 4220 2024-09-13T11:40:14.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:14.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 4 set_mon_vals no callback set 2024-09-13T11:40:14.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:14.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:14.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08d59a640 10 monclient: dump: 2024-09-13T11:40:14.980 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:14.980 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.980 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.980 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.980 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:14.981 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.981 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:14.981 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:14.981 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:14.981 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient: shutdown 2024-09-13T11:40:14.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:14.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 -- 172.21.15.5:0/1275244035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0901359f0 msgr2=0x7fb090135e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 --2- 172.21.15.5:0/1275244035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0901359f0 0x7fb090135e40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb0900a0210 tx=0x7fb08001c3e0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 -- 172.21.15.5:0/1275244035 shutdown_connections 2024-09-13T11:40:14.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 --2- 172.21.15.5:0/1275244035 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0901359f0 0x7fb090135e40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 --2- 172.21.15.5:0/1275244035 >> v2:172.21.15.5:3300/0 conn(0x7fb0900a1ae0 0x7fb0901354b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 --2- 172.21.15.5:0/1275244035 >> v2:172.21.15.39:3300/0 conn(0x7fb0900a0ea0 0x7fb0900a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 -- 172.21.15.5:0/1275244035 >> 172.21.15.5:0/1275244035 conn(0x7fb0900a2930 msgr2=0x7fb0900a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 -- 172.21.15.5:0/1275244035 shutdown_connections 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 -- 172.21.15.5:0/1275244035 wait complete. 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient: build_initial_monmap 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient: monmap: 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.982 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:14.983 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:14.983 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:14.983 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 Processor -- start 2024-09-13T11:40:14.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 -- start start 2024-09-13T11:40:14.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient: init 2024-09-13T11:40:14.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:14.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:40:14.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0900a0ea0 0x7fb090133730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0900a0ea0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:14.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0900a1ae0 0x7fb09011d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0900a1ae0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0901359f0 0x7fb09011d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0901359f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb0900a0ca0 con 0x7fb0900a0ea0 2024-09-13T11:40:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb0900a0b20 con 0x7fb0900a1ae0 2024-09-13T11:40:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fb09013f430 con 0x7fb0901359f0 2024-09-13T11:40:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb094e5b640 10 monclient(hunting): authenticate will time out at 1018.323741s 2024-09-13T11:40:14.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0900a0ea0 0x7fb090133730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0900a0ea0 0x7fb090133730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57166/0 (socket says 172.21.15.5:57166) 2024-09-13T11:40:14.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08e59c640 1 -- 172.21.15.5:0/3075751901 learned_addr learned my addr 172.21.15.5:0/3075751901 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:14.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08e59c640 10 monclient(hunting): get_auth_request con 0x7fb0900a0ea0 auth_method 0 2024-09-13T11:40:14.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.971+0000 7fb08e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08ed9d640 1 --2- 172.21.15.5:0/3075751901 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0901359f0 0x7fb09011d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08dd9b640 1 --2- 172.21.15.5:0/3075751901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0900a1ae0 0x7fb09011d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:14.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:14.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:14.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08dd9b640 10 monclient(hunting): get_auth_request con 0x7fb0900a1ae0 auth_method 0 2024-09-13T11:40:14.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08ed9d640 10 monclient(hunting): get_auth_request con 0x7fb0901359f0 auth_method 0 2024-09-13T11:40:14.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:14.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:14.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:14.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:14.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:14.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 10 monclient(hunting): handle_auth_done global_id 4245 payload 824 2024-09-13T11:40:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 1 -- 172.21.15.5:0/3075751901 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0901359f0 msgr2=0x7fb09011d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 1 --2- 172.21.15.5:0/3075751901 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0901359f0 0x7fb09011d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 1 -- 172.21.15.5:0/3075751901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0900a1ae0 msgr2=0x7fb09011d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 1 --2- 172.21.15.5:0/3075751901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0900a1ae0 0x7fb09011d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 10 monclient: _finish_hunting 0 2024-09-13T11:40:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 1 monclient: found mon.noname-a 2024-09-13T11:40:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 1 -- 172.21.15.5:0/3075751901 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09011ec30 con 0x7fb0900a0ea0 2024-09-13T11:40:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08dd9b640 1 --2- 172.21.15.5:0/3075751901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0900a1ae0 0x7fb09011d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:40:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08e59c640 1 --2- 172.21.15.5:0/3075751901 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0900a0ea0 0x7fb090133730 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb08800cf60 tx=0x7fb08800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 1 -- 172.21.15.5:0/3075751901 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08801a070 con 0x7fb0900a0ea0 2024-09-13T11:40:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 10 monclient: dump: 2024-09-13T11:40:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:14.993 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 10 monclient: _finish_auth 0 2024-09-13T11:40:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:40:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 1 -- 172.21.15.5:0/3075751901 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb08800d840 con 0x7fb0900a0ea0 2024-09-13T11:40:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 1 -- 172.21.15.5:0/3075751901 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08800dc10 con 0x7fb0900a0ea0 2024-09-13T11:40:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb094e5b640 5 monclient: authenticate success, global_id 4245 2024-09-13T11:40:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb094e5b640 10 monclient: _renew_subs 2024-09-13T11:40:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb094e5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb094e5b640 1 -- 172.21.15.5:0/3075751901 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09006dbf0 con 0x7fb0900a0ea0 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08cd99640 4 set_mon_vals no callback set 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 10 monclient: dump: 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:14.995 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb094e5b640 10 monclient: _renew_subs 2024-09-13T11:40:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb094e5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb094e5b640 1 -- 172.21.15.5:0/3075751901 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0901206f0 con 0x7fb0900a0ea0 2024-09-13T11:40:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb094e5b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb094e5b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb094e5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb094e5b640 1 -- 172.21.15.5:0/3075751901 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0901354b0 con 0x7fb0900a0ea0 2024-09-13T11:40:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 1 -- 172.21.15.5:0/3075751901 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fb08801d070 con 0x7fb0900a0ea0 2024-09-13T11:40:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 1 --2- 172.21.15.5:0/3075751901 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb07003b2a0 0x7fb07003d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb086ffd640 1 -- 172.21.15.5:0/3075751901 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fb08804f500 con 0x7fb0900a0ea0 2024-09-13T11:40:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08dd9b640 1 --2- 172.21.15.5:0/3075751901 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb07003b2a0 0x7fb07003d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08dd9b640 10 monclient: get_auth_request con 0x7fb07003b2a0 auth_method 0 2024-09-13T11:40:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.975+0000 7fb08dd9b640 1 --2- 172.21.15.5:0/3075751901 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb07003b2a0 0x7fb07003d760 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb090074ad0 tx=0x7fb078008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.979+0000 7fb086ffd640 1 -- 172.21.15.5:0/3075751901 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fb088013340 con 0x7fb0900a0ea0 2024-09-13T11:40:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.979+0000 7fb086ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:14.979+0000 7fb086ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:15.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:15.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:15.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 -- 172.21.15.5:0/3075751901 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb090069ff0 con 0x7fb0900a0ea0 2024-09-13T11:40:15.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb086ffd640 1 -- 172.21.15.5:0/3075751901 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10913 (secure 0 0 0) 0x7fb08800ddb0 con 0x7fb0900a0ea0 2024-09-13T11:40:15.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb086ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:15.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb086ffd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:15.148 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:40:15.148 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":19,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","created":"2024-09-13T11:39:53.326128+0000","modified":"2024-09-13T11:40:14.114173+0000","last_up_change":"2024-09-13T11:40:03.374234+0000","last_in_change":"2024-09-13T11:39:55.104089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:40:04.344231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:40:10.361569+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0803d65c-46ec-49b5-a45e-46bcf59ae3f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":4278099835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":4278099835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":4278099835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":4278099835}]},"public_addr":"172.21.15.5:6804/4278099835","cluster_addr":"172.21.15.5:6805/4278099835","heartbeat_back_addr":"172.21.15.5:6807/4278099835","heartbeat_front_addr":"172.21.15.5:6806/4278099835","state":["exists","up"]},{"osd":1,"uuid":"fd592a78-172c-4e1c-9f3e-c49e3e3304d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":1447734001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":1447734001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":1447734001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":1447734001}]},"public_addr":"172.21.15.5:6808/1447734001","cluster_addr":"172.21.15.5:6809/1447734001","heartbeat_back_addr":"172.21.15.5:6811/1447734001","heartbeat_front_addr":"172.21.15.5:6810/1447734001","state":["exists","up"]},{"osd":2,"uuid":"c3cec868-fd4e-472c-83b8-bc2e466671ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":1907498696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":1907498696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":1907498696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":1907498696}]},"public_addr":"172.21.15.5:6800/1907498696","cluster_addr":"172.21.15.5:6801/1907498696","heartbeat_back_addr":"172.21.15.5:6803/1907498696","heartbeat_front_addr":"172.21.15.5:6802/1907498696","state":["exists","up"]},{"osd":3,"uuid":"748f14f6-a1d6-4669-9f81-6b1cd7cd5a60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3188120659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":3188120659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":3188120659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":3188120659}]},"public_addr":"172.21.15.39:6804/3188120659","cluster_addr":"172.21.15.39:6805/3188120659","heartbeat_back_addr":"172.21.15.39:6807/3188120659","heartbeat_front_addr":"172.21.15.39:6806/3188120659","state":["exists","up"]},{"osd":4,"uuid":"f3a05376-7e83-46f9-b3e6-97b31a0942a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":4264535364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":4264535364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":4264535364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4264535364}]},"public_addr":"172.21.15.39:6800/4264535364","cluster_addr":"172.21.15.39:6801/4264535364","heartbeat_back_addr":"172.21.15.39:6803/4264535364","heartbeat_front_addr":"172.21.15.39:6802/4264535364","state":["exists","up"]},{"osd":5,"uuid":"e3b64b2d-c3e5-45d1-90a9-225892061800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1185247880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":1185247880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":1185247880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1185247880}]},"public_addr":"172.21.15.39:6808/1185247880","cluster_addr":"172.21.15.39:6809/1185247880","heartbeat_back_addr":"172.21.15.39:6811/1185247880","heartbeat_front_addr":"172.21.15.39:6810/1185247880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.244929+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.890770+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.206643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.846803+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.612582+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:02.321712+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:40:15.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 -- 172.21.15.5:0/3075751901 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb07003b2a0 msgr2=0x7fb07003d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 --2- 172.21.15.5:0/3075751901 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb07003b2a0 0x7fb07003d760 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb090074ad0 tx=0x7fb078008040 comp rx=0 tx=0).stop 2024-09-13T11:40:15.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 10 monclient: shutdown 2024-09-13T11:40:15.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 -- 172.21.15.5:0/3075751901 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0900a0ea0 msgr2=0x7fb090133730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 --2- 172.21.15.5:0/3075751901 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0900a0ea0 0x7fb090133730 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb08800cf60 tx=0x7fb08800cf90 comp rx=0 tx=0).stop 2024-09-13T11:40:15.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 -- 172.21.15.5:0/3075751901 shutdown_connections 2024-09-13T11:40:15.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 --2- 172.21.15.5:0/3075751901 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb07003b2a0 0x7fb07003d760 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 --2- 172.21.15.5:0/3075751901 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb0901359f0 0x7fb09011d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 --2- 172.21.15.5:0/3075751901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0900a1ae0 0x7fb09011d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 --2- 172.21.15.5:0/3075751901 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0900a0ea0 0x7fb090133730 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 -- 172.21.15.5:0/3075751901 >> 172.21.15.5:0/3075751901 conn(0x7fb0900a2930 msgr2=0x7fb0901347b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 -- 172.21.15.5:0/3075751901 shutdown_connections 2024-09-13T11:40:15.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.147+0000 7fb094e5b640 1 -- 172.21.15.5:0/3075751901 wait complete. 2024-09-13T11:40:15.165 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T11:40:15.166 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:40:15.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.283+0000 7fd4ad1da640 10 monclient: get_monmap_and_config 2024-09-13T11:40:15.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.283+0000 7fd4ad1da640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.283+0000 7fd4ad1da640 10 monclient: monmap: 2024-09-13T11:40:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:15.287511+0000 2024-09-13T11:40:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:15.287511+0000 2024-09-13T11:40:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:15.287 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:15.287 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.283+0000 7fd4ad1da640 1 Processor -- start 2024-09-13T11:40:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.283+0000 7fd4ad1da640 1 -- start start 2024-09-13T11:40:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.283+0000 7fd4ad1da640 10 monclient: init 2024-09-13T11:40:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-13T11:40:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4a8134810 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd4a8137660 0x7fd4a8139a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 10 monclient(hunting): picked mon.noname-c con 0x7fd4a8137660 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4a8139f90 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4a809ce60 con 0x7fd4a8139f90 2024-09-13T11:40:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4a809eeb0 con 0x7fd4a8134810 2024-09-13T11:40:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fd4a80a0950 con 0x7fd4a8137660 2024-09-13T11:40:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 10 monclient(hunting): authenticate will time out at 1018.636335s 2024-09-13T11:40:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd4a8139f90 0x7fd4a813c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36786/0 (socket says 172.21.15.5:36786) 2024-09-13T11:40:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 1 -- 172.21.15.5:0/1791143250 learned_addr learned my addr 172.21.15.5:0/1791143250 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 1 --2- 172.21.15.5:0/1791143250 >> v2:172.21.15.39:3300/0 conn(0x7fd4a8137660 0x7fd4a8139a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): get_auth_request con 0x7fd4a8137660 auth_method 0 2024-09-13T11:40:15.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2671622427 0 0) 0x7fd4a809ce60 con 0x7fd4a8139f90 2024-09-13T11:40:15.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient(hunting): _negotiate 2024-09-13T11:40:15.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient(hunting): my global_id is 4248 2024-09-13T11:40:15.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd494003530 con 0x7fd4a8139f90 2024-09-13T11:40:15.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 490340511 0 0) 0x7fd494003530 con 0x7fd4a8139f90 2024-09-13T11:40:15.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 >> v2:172.21.15.39:3300/0 conn(0x7fd4a8137660 msgr2=0x7fd4a8139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:15.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 --2- 172.21.15.5:0/1791143250 >> v2:172.21.15.39:3300/0 conn(0x7fd4a8137660 0x7fd4a8139a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 >> v1:172.21.15.39:6789/0 conn(0x7fd4a8134810 legacy=0x7fd4a8134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:40:15.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a809e1e0 con 0x7fd4a8139f90 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3199362888 0 0) 0x7fd498001700 con 0x7fd4a8139f90 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient: dump: 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 >> v1:172.21.15.5:6789/0 conn(0x7fd4a8139f90 legacy=0x7fd4a813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 --2- 172.21.15.5:0/1791143250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd494003a60 0x7fd494003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient(hunting): picked mon.b con 0x7fd494003a60 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 --2- 172.21.15.5:0/1791143250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd494004810 0x7fd494004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient(hunting): picked mon.c con 0x7fd494004810 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 --2- 172.21.15.5:0/1791143250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4940053d0 0x7fd49400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient(hunting): picked mon.a con 0x7fd4940053d0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd49400c210 con 0x7fd4940053d0 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd49400c3c0 con 0x7fd494003a60 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fd49400c5c0 con 0x7fd494004810 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a5d9b640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 1 --2- 172.21.15.5:0/1791143250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd494004810 0x7fd494004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 1 --2- 172.21.15.5:0/1791143250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4940053d0 0x7fd49400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 10 monclient(hunting): get_auth_request con 0x7fd4940053d0 auth_method 0 2024-09-13T11:40:15.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): get_auth_request con 0x7fd494004810 auth_method 0 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a6d9d640 1 --2- 172.21.15.5:0/1791143250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd494003a60 0x7fd494003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a6d9d640 10 monclient(hunting): get_auth_request con 0x7fd494003a60 auth_method 0 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 10 monclient(hunting): handle_auth_done global_id 4248 payload 293 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 1 -- 172.21.15.5:0/1791143250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd494004810 msgr2=0x7fd494004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 1 --2- 172.21.15.5:0/1791143250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd494004810 0x7fd494004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 1 -- 172.21.15.5:0/1791143250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd494003a60 msgr2=0x7fd494003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 1 --2- 172.21.15.5:0/1791143250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd494003a60 0x7fd494003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 1 monclient: found mon.a 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 1 -- 172.21.15.5:0/1791143250 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd494003530 con 0x7fd4940053d0 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 5 monclient: authenticate success, global_id 4248 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4ad1da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 1 -- 172.21.15.5:0/1791143250 reap_dead start 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.287+0000 7fd4a759e640 1 --2- 172.21.15.5:0/1791143250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4940053d0 0x7fd49400b9d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c210 tx=0x7fd49801cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48198971e8ba6ec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498007d20 con 0x7fd4940053d0 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a5d9b640 10 monclient: dump: 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.306 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4980a9070 con 0x7fd4940053d0 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a5d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a5d9b640 1 -- 172.21.15.5:0/1791143250 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498005d60 con 0x7fd4940053d0 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 4 set_mon_vals no callback set 2024-09-13T11:40:15.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a5d9b640 10 monclient: dump: 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: shutdown 2024-09-13T11:40:15.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 -- 172.21.15.5:0/1791143250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4940053d0 msgr2=0x7fd49400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 --2- 172.21.15.5:0/1791143250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4940053d0 0x7fd49400b9d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c210 tx=0x7fd49801cba0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 -- 172.21.15.5:0/1791143250 shutdown_connections 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 --2- 172.21.15.5:0/1791143250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4940053d0 0x7fd49400b9d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 -- 172.21.15.5:0/1791143250 >> 172.21.15.5:0/1791143250 conn(0x7fd4a80a2930 msgr2=0x7fd4a80a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 -- 172.21.15.5:0/1791143250 shutdown_connections 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 -- 172.21.15.5:0/1791143250 wait complete. 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: monmap: 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:15.309 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 Processor -- start 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 -- start start 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: init 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4940053d0 0x7fd4a811c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient(hunting): picked mon.noname-b con 0x7fd4940053d0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a8072510 0x7fd4a8133610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient(hunting): picked mon.noname-a con 0x7fd4a8072510 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a8133b50 0x7fd4a81cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient(hunting): picked mon.noname-c con 0x7fd4a8133b50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a8134f80 con 0x7fd4a8072510 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a8134e00 con 0x7fd4940053d0 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fd4a8135100 con 0x7fd4a8133b50 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient(hunting): authenticate will time out at 1018.642170s 2024-09-13T11:40:15.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a8133b50 0x7fd4a81cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a8133b50 0x7fd4a81cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59108/0 (socket says 172.21.15.5:59108) 2024-09-13T11:40:15.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 1 -- 172.21.15.5:0/2056550018 learned_addr learned my addr 172.21.15.5:0/2056550018 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 10 monclient(hunting): get_auth_request con 0x7fd4a8133b50 auth_method 0 2024-09-13T11:40:15.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a8072510 0x7fd4a8133610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a659c640 10 monclient(hunting): get_auth_request con 0x7fd4a8072510 auth_method 0 2024-09-13T11:40:15.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a6d9d640 1 --2- 172.21.15.5:0/2056550018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4940053d0 0x7fd4a811c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a6d9d640 10 monclient(hunting): get_auth_request con 0x7fd4940053d0 auth_method 0 2024-09-13T11:40:15.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2024-09-13T11:40:15.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 1 -- 172.21.15.5:0/2056550018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4940053d0 msgr2=0x7fd4a811c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 1 --2- 172.21.15.5:0/2056550018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4940053d0 0x7fd4a811c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 1 -- 172.21.15.5:0/2056550018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a8072510 msgr2=0x7fd4a8133610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 1 --2- 172.21.15.5:0/2056550018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a8072510 0x7fd4a8133610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 1 monclient: found mon.noname-c 2024-09-13T11:40:15.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 1 -- 172.21.15.5:0/2056550018 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a8074b90 con 0x7fd4a8133b50 2024-09-13T11:40:15.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a6d9d640 1 --2- 172.21.15.5:0/2056550018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4940053d0 0x7fd4a811c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:40:15.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a659c640 1 --2- 172.21.15.5:0/2056550018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a8072510 0x7fd4a8133610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:15.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a759e640 1 --2- 172.21.15.5:0/2056550018 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a8133b50 0x7fd4a81cd640 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd498000c10 tx=0x7fd494005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 1 -- 172.21.15.5:0/2056550018 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498001060 con 0x7fd4a8133b50 2024-09-13T11:40:15.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:15.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 10 monclient: dump: 2024-09-13T11:40:15.315 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.315 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.315 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.315 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.315 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.315 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.315 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.316 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.316 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.316 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 1 -- 172.21.15.5:0/2056550018 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd498005b00 con 0x7fd4a8133b50 2024-09-13T11:40:15.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 1 -- 172.21.15.5:0/2056550018 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4980047f0 con 0x7fd4a8133b50 2024-09-13T11:40:15.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 5 monclient: authenticate success, global_id 4226 2024-09-13T11:40:15.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: _renew_subs 2024-09-13T11:40:15.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4a559a640 4 set_mon_vals no callback set 2024-09-13T11:40:15.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 -- 172.21.15.5:0/2056550018 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a811c2b0 con 0x7fd4a8133b50 2024-09-13T11:40:15.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:15.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd49f7fe640 10 monclient: dump: 2024-09-13T11:40:15.317 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.317 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.317 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.317 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.317 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.318 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.318 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.318 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.318 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.318 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: _renew_subs 2024-09-13T11:40:15.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 -- 172.21.15.5:0/2056550018 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4a811dfa0 con 0x7fd4a8133b50 2024-09-13T11:40:15.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:15.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:15.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.291+0000 7fd4ad1da640 1 -- 172.21.15.5:0/2056550018 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4a806dbf0 con 0x7fd4a8133b50 2024-09-13T11:40:15.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.295+0000 7fd49f7fe640 1 -- 172.21.15.5:0/2056550018 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fd4980b5030 con 0x7fd4a8133b50 2024-09-13T11:40:15.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.295+0000 7fd49f7fe640 1 --2- 172.21.15.5:0/2056550018 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd48803b2d0 0x7fd48803d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.295+0000 7fd49f7fe640 1 -- 172.21.15.5:0/2056550018 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fd4980e6e10 con 0x7fd4a8133b50 2024-09-13T11:40:15.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.295+0000 7fd4a6d9d640 1 --2- 172.21.15.5:0/2056550018 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd48803b2d0 0x7fd48803d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.295+0000 7fd4a6d9d640 10 monclient: get_auth_request con 0x7fd48803b2d0 auth_method 0 2024-09-13T11:40:15.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.295+0000 7fd4a6d9d640 1 --2- 172.21.15.5:0/2056550018 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd48803b2d0 0x7fd48803d790 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0000f90 tx=0x7fd4a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.299+0000 7fd49f7fe640 1 -- 172.21.15.5:0/2056550018 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fd4980bc060 con 0x7fd4a8133b50 2024-09-13T11:40:15.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.299+0000 7fd49f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:15.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.299+0000 7fd49f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:15.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.471+0000 7fd4ad1da640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:15.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.471+0000 7fd4ad1da640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:15.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.471+0000 7fd4ad1da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.471+0000 7fd4ad1da640 1 -- 172.21.15.5:0/2056550018 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4a809ce60 con 0x7fd4a8133b50 2024-09-13T11:40:15.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd49f7fe640 1 -- 172.21.15.5:0/2056550018 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10913 (secure 0 0 0) 0x7fd4a8074b90 con 0x7fd4a8133b50 2024-09-13T11:40:15.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd49f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:15.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd49f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:15.475 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:40:15.475 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":19,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","created":"2024-09-13T11:39:53.326128+0000","modified":"2024-09-13T11:40:14.114173+0000","last_up_change":"2024-09-13T11:40:03.374234+0000","last_in_change":"2024-09-13T11:39:55.104089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:40:04.344231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:40:10.361569+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0803d65c-46ec-49b5-a45e-46bcf59ae3f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":4278099835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":4278099835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":4278099835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":4278099835}]},"public_addr":"172.21.15.5:6804/4278099835","cluster_addr":"172.21.15.5:6805/4278099835","heartbeat_back_addr":"172.21.15.5:6807/4278099835","heartbeat_front_addr":"172.21.15.5:6806/4278099835","state":["exists","up"]},{"osd":1,"uuid":"fd592a78-172c-4e1c-9f3e-c49e3e3304d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":1447734001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":1447734001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":1447734001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":1447734001}]},"public_addr":"172.21.15.5:6808/1447734001","cluster_addr":"172.21.15.5:6809/1447734001","heartbeat_back_addr":"172.21.15.5:6811/1447734001","heartbeat_front_addr":"172.21.15.5:6810/1447734001","state":["exists","up"]},{"osd":2,"uuid":"c3cec868-fd4e-472c-83b8-bc2e466671ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":1907498696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":1907498696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":1907498696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":1907498696}]},"public_addr":"172.21.15.5:6800/1907498696","cluster_addr":"172.21.15.5:6801/1907498696","heartbeat_back_addr":"172.21.15.5:6803/1907498696","heartbeat_front_addr":"172.21.15.5:6802/1907498696","state":["exists","up"]},{"osd":3,"uuid":"748f14f6-a1d6-4669-9f81-6b1cd7cd5a60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3188120659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":3188120659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":3188120659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":3188120659}]},"public_addr":"172.21.15.39:6804/3188120659","cluster_addr":"172.21.15.39:6805/3188120659","heartbeat_back_addr":"172.21.15.39:6807/3188120659","heartbeat_front_addr":"172.21.15.39:6806/3188120659","state":["exists","up"]},{"osd":4,"uuid":"f3a05376-7e83-46f9-b3e6-97b31a0942a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":4264535364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":4264535364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":4264535364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4264535364}]},"public_addr":"172.21.15.39:6800/4264535364","cluster_addr":"172.21.15.39:6801/4264535364","heartbeat_back_addr":"172.21.15.39:6803/4264535364","heartbeat_front_addr":"172.21.15.39:6802/4264535364","state":["exists","up"]},{"osd":5,"uuid":"e3b64b2d-c3e5-45d1-90a9-225892061800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1185247880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":1185247880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":1185247880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1185247880}]},"public_addr":"172.21.15.39:6808/1185247880","cluster_addr":"172.21.15.39:6809/1185247880","heartbeat_back_addr":"172.21.15.39:6811/1185247880","heartbeat_front_addr":"172.21.15.39:6810/1185247880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.244929+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.890770+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.206643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.846803+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.612582+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:02.321712+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:40:15.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 1 -- 172.21.15.5:0/2056550018 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd48803b2d0 msgr2=0x7fd48803d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 1 --2- 172.21.15.5:0/2056550018 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd48803b2d0 0x7fd48803d790 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0000f90 tx=0x7fd4a0009040 comp rx=0 tx=0).stop 2024-09-13T11:40:15.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 10 monclient: shutdown 2024-09-13T11:40:15.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 1 -- 172.21.15.5:0/2056550018 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a8133b50 msgr2=0x7fd4a81cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 1 --2- 172.21.15.5:0/2056550018 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a8133b50 0x7fd4a81cd640 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd498000c10 tx=0x7fd494005840 comp rx=0 tx=0).stop 2024-09-13T11:40:15.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 1 -- 172.21.15.5:0/2056550018 shutdown_connections 2024-09-13T11:40:15.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 1 --2- 172.21.15.5:0/2056550018 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd48803b2d0 0x7fd48803d790 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 1 --2- 172.21.15.5:0/2056550018 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4a8133b50 0x7fd4a81cd640 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 1 --2- 172.21.15.5:0/2056550018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4a8072510 0x7fd4a8133610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 1 --2- 172.21.15.5:0/2056550018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4940053d0 0x7fd4a811c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 1 -- 172.21.15.5:0/2056550018 >> 172.21.15.5:0/2056550018 conn(0x7fd4a80a2930 msgr2=0x7fd4a813cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 1 -- 172.21.15.5:0/2056550018 shutdown_connections 2024-09-13T11:40:15.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.475+0000 7fd4ad1da640 1 -- 172.21.15.5:0/2056550018 wait complete. 2024-09-13T11:40:15.494 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T11:40:15.494 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T11:40:15.495 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T11:40:15.495 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T11:40:15.495 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T11:40:15.496 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T11:40:15.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.607+0000 7fc2922a6640 10 monclient: get_monmap_and_config 2024-09-13T11:40:15.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.607+0000 7fc2922a6640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.607+0000 7fc2922a6640 10 monclient: monmap: 2024-09-13T11:40:15.611 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.611 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.611 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:15.612746+0000 2024-09-13T11:40:15.611 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:15.612746+0000 2024-09-13T11:40:15.611 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.611 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.611 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:15.611 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:15.611 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:15.611 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:15.612 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:15.612 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 1 Processor -- start 2024-09-13T11:40:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 1 -- start start 2024-09-13T11:40:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 10 monclient: init 2024-09-13T11:40:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-13T11:40:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc28c137930 0x7fc28c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 10 monclient(hunting): picked mon.noname-c con 0x7fc28c137930 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc28c137d10 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc28c13cd80 0x7fc28c13f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 10 monclient(hunting): picked mon.noname-a con 0x7fc28c13cd80 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:15.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc28c0a5a80 con 0x7fc28c137d10 2024-09-13T11:40:15.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fc28c0a5fe0 con 0x7fc28c13cd80 2024-09-13T11:40:15.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fc28c09c540 con 0x7fc28c137930 2024-09-13T11:40:15.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc2922a6640 10 monclient(hunting): authenticate will time out at 1018.961172s 2024-09-13T11:40:15.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc28b7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fc28c137d10 0x7fc28c1380f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36790/0 (socket says 172.21.15.5:36790) 2024-09-13T11:40:15.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc28b7fe640 1 -- 172.21.15.5:0/1442039495 learned_addr learned my addr 172.21.15.5:0/1442039495 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:15.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 1 --2- 172.21.15.5:0/1442039495 >> v2:172.21.15.5:3300/0 conn(0x7fc28c13cd80 0x7fc28c13f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 10 monclient(hunting): get_auth_request con 0x7fc28c13cd80 auth_method 0 2024-09-13T11:40:15.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc28bfff640 1 --2- 172.21.15.5:0/1442039495 >> v2:172.21.15.39:3300/0 conn(0x7fc28c137930 0x7fc28c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc28bfff640 10 monclient(hunting): get_auth_request con 0x7fc28c137930 auth_method 0 2024-09-13T11:40:15.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc28bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 10 monclient(hunting): handle_auth_done global_id 4254 payload 274 2024-09-13T11:40:15.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc28affd640 1 -- 172.21.15.5:0/1442039495 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460138245 0 0) 0x7fc28c0a5a80 con 0x7fc28c137d10 2024-09-13T11:40:15.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 1 -- 172.21.15.5:0/1442039495 >> v2:172.21.15.39:3300/0 conn(0x7fc28c137930 msgr2=0x7fc28c09cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:15.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 1 --2- 172.21.15.5:0/1442039495 >> v2:172.21.15.39:3300/0 conn(0x7fc28c137930 0x7fc28c09cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 1 -- 172.21.15.5:0/1442039495 >> v1:172.21.15.5:6789/0 conn(0x7fc28c137d10 legacy=0x7fc28c1380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 1 monclient: found mon.noname-a 2024-09-13T11:40:15.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.611+0000 7fc290843640 1 -- 172.21.15.5:0/1442039495 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28c09bef0 con 0x7fc28c13cd80 2024-09-13T11:40:15.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:40:15.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc290843640 1 --2- 172.21.15.5:0/1442039495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc28c13cd80 0x7fc28c13f170 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0a7cb0 tx=0x7fc28401c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7e34f411952d87 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 10 monclient: get_monmap_and_config 2024-09-13T11:40:15.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 10 monclient: monmap: 2024-09-13T11:40:15.620 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.620 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.620 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:15.618147+0000 2024-09-13T11:40:15.620 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:15.618147+0000 2024-09-13T11:40:15.620 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.620 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.620 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:15.620 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:15.620 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:15.620 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:15.621 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:15.621 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 1 -- 172.21.15.5:0/1442039495 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2840b2070 con 0x7fc28c13cd80 2024-09-13T11:40:15.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:15.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 10 monclient: dump: 2024-09-13T11:40:15.622 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.622 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.622 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.622 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.622 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.622 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.622 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.622 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.622 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.622 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 5 monclient: authenticate success, global_id 4254 2024-09-13T11:40:15.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:15.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 1 -- 172.21.15.5:0/1442039495 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2840a57c0 con 0x7fc28c13cd80 2024-09-13T11:40:15.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 1 -- 172.21.15.5:0/1442039495 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2840a5b90 con 0x7fc28c13cd80 2024-09-13T11:40:15.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc28affd640 10 monclient: dump: 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:15.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 4 set_mon_vals no callback set 2024-09-13T11:40:15.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 1 Processor -- start 2024-09-13T11:40:15.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 10 monclient: shutdown 2024-09-13T11:40:15.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 1 -- 172.21.15.5:0/1442039495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc28c13cd80 msgr2=0x7fc28c13f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 1 --2- 172.21.15.5:0/1442039495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc28c13cd80 0x7fc28c13f170 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0a7cb0 tx=0x7fc28401c810 comp rx=0 tx=0).stop 2024-09-13T11:40:15.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 1 -- start start 2024-09-13T11:40:15.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 10 monclient: init 2024-09-13T11:40:15.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 1 -- 172.21.15.5:0/1442039495 shutdown_connections 2024-09-13T11:40:15.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 1 --2- 172.21.15.5:0/1442039495 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc28c13cd80 0x7fc28c13f170 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 1 --2- 172.21.15.5:0/1442039495 >> v2:172.21.15.39:3300/0 conn(0x7fc28c137930 0x7fc28c09cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 1 -- 172.21.15.5:0/1442039495 >> 172.21.15.5:0/1442039495 conn(0x7fc28c0a6c70 msgr2=0x7fc28c0a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 1 -- 172.21.15.5:0/1442039495 shutdown_connections 2024-09-13T11:40:15.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 1 -- 172.21.15.5:0/1442039495 wait complete. 2024-09-13T11:40:15.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-13T11:40:15.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2544137930 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:15.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2544138420 0x7f254409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 10 monclient(hunting): picked mon.noname-c con 0x7f2544138420 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:15.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f254413cd80 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:15.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25440a5a80 con 0x7f254413cd80 2024-09-13T11:40:15.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f254409bef0 con 0x7f2544137930 2024-09-13T11:40:15.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f25440a5fe0 con 0x7f2544138420 2024-09-13T11:40:15.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254bef4640 10 monclient(hunting): authenticate will time out at 1018.967157s 2024-09-13T11:40:15.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254a491640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f254413cd80 0x7f254413f170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36796/0 (socket says 172.21.15.5:36796) 2024-09-13T11:40:15.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254a491640 1 -- 172.21.15.5:0/2824857789 learned_addr learned my addr 172.21.15.5:0/2824857789 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:15.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 10 monclient: monmap: 2024-09-13T11:40:15.630 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.630 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:15.630 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:15.630 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:15.630 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.630 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.630 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:15.630 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:15.630 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:15.630 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254948f640 1 --2- 172.21.15.5:0/2824857789 >> v2:172.21.15.39:3300/0 conn(0x7f2544138420 0x7f254409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2146441843 0 0) 0x7f25440a5a80 con 0x7f254413cd80 2024-09-13T11:40:15.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254948f640 10 monclient(hunting): get_auth_request con 0x7f2544138420 auth_method 0 2024-09-13T11:40:15.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254948f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254948f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254948f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f2548c8e640 10 monclient(hunting): _negotiate 2024-09-13T11:40:15.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f2548c8e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f2548c8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f2548c8e640 10 monclient(hunting): my global_id is 4260 2024-09-13T11:40:15.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2530003530 con 0x7f254413cd80 2024-09-13T11:40:15.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254948f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254948f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7f254948f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.615+0000 7fc2922a6640 1 Processor -- start 2024-09-13T11:40:15.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1256984419 0 0) 0x7f2530003530 con 0x7f254413cd80 2024-09-13T11:40:15.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 >> v2:172.21.15.39:3300/0 conn(0x7f2544138420 msgr2=0x7f254409cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:15.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 --2- 172.21.15.5:0/2824857789 >> v2:172.21.15.39:3300/0 conn(0x7f2544138420 0x7f254409cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 >> v1:172.21.15.39:6789/0 conn(0x7f2544137930 legacy=0x7f2544137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 1 -- start start 2024-09-13T11:40:15.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient: init 2024-09-13T11:40:15.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:40:15.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-13T11:40:15.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254409c1d0 con 0x7f254413cd80 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254bef4640 5 monclient: authenticate success, global_id 4260 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3199362888 0 0) 0x7f253c001a80 con 0x7f254413cd80 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254bef4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient: dump: 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.635 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.636 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.636 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.636 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.636 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.636 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-13T11:40:15.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 >> v1:172.21.15.5:6789/0 conn(0x7f254413cd80 legacy=0x7f254413f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:15.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 --2- 172.21.15.5:0/2824857789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2530003ab0 0x7f2530003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient(hunting): picked mon.c con 0x7f2530003ab0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 --2- 172.21.15.5:0/2824857789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2530004810 0x7f2530004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient(hunting): picked mon.b con 0x7f2530004810 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:15.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 --2- 172.21.15.5:0/2824857789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25300053d0 0x7f253000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient(hunting): picked mon.a con 0x7f25300053d0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:15.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f253000c260 con 0x7f25300053d0 2024-09-13T11:40:15.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f253000c410 con 0x7f2530004810 2024-09-13T11:40:15.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f2530014e20 con 0x7f2530003ab0 2024-09-13T11:40:15.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f2548c8e640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:40:15.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254bef4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:15.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 1 --2- 172.21.15.5:0/2824857789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25300053d0 0x7f253000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 10 monclient(hunting): get_auth_request con 0x7f25300053d0 auth_method 0 2024-09-13T11:40:15.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:15.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254948f640 1 --2- 172.21.15.5:0/2824857789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2530004810 0x7f2530004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:40:15.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254948f640 10 monclient(hunting): get_auth_request con 0x7f2530004810 auth_method 0 2024-09-13T11:40:15.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254948f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254948f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254948f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:15.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:15.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc28c137930 0x7fc28c1361c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 10 monclient(hunting): handle_auth_done global_id 4260 payload 293 2024-09-13T11:40:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 1 -- 172.21.15.5:0/2824857789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2530003ab0 msgr2=0x7f2530003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 1 --2- 172.21.15.5:0/2824857789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2530003ab0 0x7f2530003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 1 -- 172.21.15.5:0/2824857789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2530004810 msgr2=0x7f2530004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 1 --2- 172.21.15.5:0/2824857789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2530004810 0x7f2530004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 1 monclient: found mon.a 2024-09-13T11:40:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 1 -- 172.21.15.5:0/2824857789 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2530003530 con 0x7f25300053d0 2024-09-13T11:40:15.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient(hunting): picked mon.noname-c con 0x7fc28c137930 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc28c137d10 0x7fc28c136700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28bfff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc28c137930 0x7fc28c1361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28bfff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc28c137930 0x7fc28c1361c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59124/0 (socket says 172.21.15.5:59124) 2024-09-13T11:40:15.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient(hunting): picked mon.noname-b con 0x7fc28c137d10 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:15.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28bfff640 1 -- 172.21.15.5:0/1511412036 learned_addr learned my addr 172.21.15.5:0/1511412036 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 1 --2- 172.21.15.5:0/1511412036 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc28c13cd80 0x7fc28c1399d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 1 --2- 172.21.15.5:0/1511412036 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc28c13cd80 0x7fc28c1399d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28b7fe640 1 --2- 172.21.15.5:0/1511412036 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc28c137d10 0x7fc28c136700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient(hunting): picked mon.noname-a con 0x7fc28c13cd80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc28c142350 con 0x7fc28c13cd80 2024-09-13T11:40:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc28c1421d0 con 0x7fc28c137d10 2024-09-13T11:40:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fc28c1424d0 con 0x7fc28c137930 2024-09-13T11:40:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient(hunting): authenticate will time out at 1018.969809s 2024-09-13T11:40:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28bfff640 10 monclient(hunting): get_auth_request con 0x7fc28c137930 auth_method 0 2024-09-13T11:40:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 10 monclient(hunting): get_auth_request con 0x7fc28c13cd80 auth_method 0 2024-09-13T11:40:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28b7fe640 10 monclient(hunting): get_auth_request con 0x7fc28c137d10 auth_method 0 2024-09-13T11:40:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2024-09-13T11:40:15.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 1 -- 172.21.15.5:0/1511412036 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc28c137930 msgr2=0x7fc28c1361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 1 --2- 172.21.15.5:0/1511412036 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc28c137930 0x7fc28c1361c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 1 -- 172.21.15.5:0/1511412036 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc28c137d10 msgr2=0x7fc28c136700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 1 --2- 172.21.15.5:0/1511412036 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc28c137d10 0x7fc28c136700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 1 monclient: found mon.noname-a 2024-09-13T11:40:15.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 1 -- 172.21.15.5:0/1511412036 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28c120f60 con 0x7fc28c13cd80 2024-09-13T11:40:15.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc290843640 1 --2- 172.21.15.5:0/1511412036 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc28c13cd80 0x7fc28c1399d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc2840ac000 tx=0x7fc2840195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc28bfff640 1 --2- 172.21.15.5:0/1511412036 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc28c137930 0x7fc28c1361c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:15.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 1 -- 172.21.15.5:0/1511412036 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2840b2050 con 0x7fc28c13cd80 2024-09-13T11:40:15.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 10 monclient: dump: 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 1 -- 172.21.15.5:0/1511412036 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2840b3040 con 0x7fc28c13cd80 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 1 -- 172.21.15.5:0/1511412036 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284003ba0 con 0x7fc28c13cd80 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc288ff9640 10 monclient: dump: 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.652 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.652 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.652 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 5 monclient: authenticate success, global_id 4263 2024-09-13T11:40:15.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient: _renew_subs 2024-09-13T11:40:15.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc28c121240 con 0x7fc28c13cd80 2024-09-13T11:40:15.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient: _renew_subs 2024-09-13T11:40:15.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc28c0a1360 con 0x7fc28c13cd80 2024-09-13T11:40:15.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.619+0000 7f254a491640 1 -- 172.21.15.5:0/2824857789 reap_dead start 2024-09-13T11:40:15.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254a491640 1 --2- 172.21.15.5:0/2824857789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25300053d0 0x7f253000b9d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f253c00a7d0 tx=0x7f253c01d220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a39a66a34d30740 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc2922a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc28c09cc90 con 0x7fc28c13cd80 2024-09-13T11:40:15.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c00a3f0 con 0x7f25300053d0 2024-09-13T11:40:15.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2548c8e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2548c8e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:15.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2548c8e640 10 monclient: dump: 2024-09-13T11:40:15.655 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.655 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.655 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.655 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.655 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.655 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.655 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.655 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f253c0ac620 con 0x7f25300053d0 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2548c8e640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 4 set_mon_vals no callback set 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2548c8e640 1 -- 172.21.15.5:0/2824857789 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c0aca10 con 0x7f25300053d0 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2548c8e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2548c8e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2548c8e640 10 monclient: dump: 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.656 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.657 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.657 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.657 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.657 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.657 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.657 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.657 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.657 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient: shutdown 2024-09-13T11:40:15.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 -- 172.21.15.5:0/2824857789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25300053d0 msgr2=0x7f253000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 --2- 172.21.15.5:0/2824857789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25300053d0 0x7f253000b9d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f253c00a7d0 tx=0x7f253c01d220 comp rx=0 tx=0).stop 2024-09-13T11:40:15.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 -- 172.21.15.5:0/2824857789 shutdown_connections 2024-09-13T11:40:15.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 --2- 172.21.15.5:0/2824857789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25300053d0 0x7f253000b9d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 -- 172.21.15.5:0/2824857789 >> 172.21.15.5:0/2824857789 conn(0x7f25440a6c70 msgr2=0x7f25440a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 -- 172.21.15.5:0/2824857789 shutdown_connections 2024-09-13T11:40:15.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 -- 172.21.15.5:0/2824857789 wait complete. 2024-09-13T11:40:15.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc28a7fc640 4 set_mon_vals no callback set 2024-09-13T11:40:15.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc288ff9640 1 -- 172.21.15.5:0/1511412036 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fc2840ad650 con 0x7fc28c13cd80 2024-09-13T11:40:15.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc288ff9640 1 --2- 172.21.15.5:0/1511412036 >> v2:172.21.15.39:6812/1340838781 conn(0x7fc27803b320 0x7fc27803d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc288ff9640 1 -- 172.21.15.5:0/1511412036 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fc2840e9600 con 0x7fc28c13cd80 2024-09-13T11:40:15.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc288ff9640 1 --2- 172.21.15.5:0/1511412036 >> v2:172.21.15.5:6804/4278099835 conn(0x7fc278040eb0 0x7fc2780432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc288ff9640 1 -- 172.21.15.5:0/1511412036 --> v2:172.21.15.5:6804/4278099835 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2840a57c0 con 0x7fc278040eb0 2024-09-13T11:40:15.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc288ff9640 1 -- 172.21.15.5:0/1511412036 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc2840f0050 con 0x7fc28c13cd80 2024-09-13T11:40:15.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc288ff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-13T11:40:15.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient: monmap: 2024-09-13T11:40:15.660 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.660 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:15.660 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:15.660 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:15.660 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.660 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.660 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:15.661 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:15.661 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:15.661 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc28b7fe640 1 --2- 172.21.15.5:0/1511412036 >> v2:172.21.15.5:6804/4278099835 conn(0x7fc278040eb0 0x7fc2780432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc28b7fe640 10 monclient: get_auth_request con 0x7fc278040eb0 auth_method 0 2024-09-13T11:40:15.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 Processor -- start 2024-09-13T11:40:15.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 -- start start 2024-09-13T11:40:15.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient: init 2024-09-13T11:40:15.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc28bfff640 1 --2- 172.21.15.5:0/1511412036 >> v2:172.21.15.39:6812/1340838781 conn(0x7fc27803b320 0x7fc27803d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7fc28bfff640 10 monclient: get_auth_request con 0x7fc27803b320 auth_method 0 2024-09-13T11:40:15.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:40:15.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25300053d0 0x7f2544136380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient(hunting): picked mon.noname-c con 0x7f25300053d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25441368c0 0x7f254413a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient(hunting): picked mon.noname-a con 0x7f25441368c0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:15.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2544136d40 0x7f254413aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient(hunting): picked mon.noname-b con 0x7f2544136d40 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:15.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f254409d020 con 0x7f25441368c0 2024-09-13T11:40:15.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f254409cea0 con 0x7f2544136d40 2024-09-13T11:40:15.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f254409c500 con 0x7f25300053d0 2024-09-13T11:40:15.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254bef4640 10 monclient(hunting): authenticate will time out at 1018.975200s 2024-09-13T11:40:15.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25300053d0 0x7f2544136380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25300053d0 0x7f2544136380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59130/0 (socket says 172.21.15.5:59130) 2024-09-13T11:40:15.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 1 -- 172.21.15.5:0/865163143 learned_addr learned my addr 172.21.15.5:0/865163143 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 10 monclient(hunting): get_auth_request con 0x7f25300053d0 auth_method 0 2024-09-13T11:40:15.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254948f640 1 --2- 172.21.15.5:0/865163143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25441368c0 0x7f254413a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254948f640 10 monclient(hunting): get_auth_request con 0x7f25441368c0 auth_method 0 2024-09-13T11:40:15.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254948f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254948f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254948f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254a491640 1 --2- 172.21.15.5:0/865163143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2544136d40 0x7f254413aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254a491640 10 monclient(hunting): get_auth_request con 0x7f2544136d40 auth_method 0 2024-09-13T11:40:15.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254a491640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254a491640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254a491640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254948f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254948f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254948f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254a491640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254a491640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f254a491640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-09-13T11:40:15.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 1 -- 172.21.15.5:0/865163143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2544136d40 msgr2=0x7f254413aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 1 --2- 172.21.15.5:0/865163143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2544136d40 0x7f254413aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 1 -- 172.21.15.5:0/865163143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25441368c0 msgr2=0x7f254413a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 1 --2- 172.21.15.5:0/865163143 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25441368c0 0x7f254413a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 1 monclient: found mon.noname-c 2024-09-13T11:40:15.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.623+0000 7f2549c90640 1 -- 172.21.15.5:0/865163143 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25440a1360 con 0x7f25300053d0 2024-09-13T11:40:15.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7fc28b7fe640 1 --2- 172.21.15.5:0/1511412036 >> v2:172.21.15.5:6804/4278099835 conn(0x7fc278040eb0 0x7fc2780432f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7fc28bfff640 1 --2- 172.21.15.5:0/1511412036 >> v2:172.21.15.39:6812/1340838781 conn(0x7fc27803b320 0x7fc27803d7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc27c009920 tx=0x7fc27c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f2549c90640 1 --2- 172.21.15.5:0/865163143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25300053d0 0x7f2544136380 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f254000c520 tx=0x7f254000c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 1 -- 172.21.15.5:0/865163143 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540007020 con 0x7f25300053d0 2024-09-13T11:40:15.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:15.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 10 monclient: dump: 2024-09-13T11:40:15.673 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.673 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.673 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.673 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.673 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.673 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.673 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.674 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.674 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.674 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254bef4640 5 monclient: authenticate success, global_id 4229 2024-09-13T11:40:15.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 1 -- 172.21.15.5:0/865163143 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25400071c0 con 0x7f25300053d0 2024-09-13T11:40:15.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 1 -- 172.21.15.5:0/865163143 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540007590 con 0x7f25300053d0 2024-09-13T11:40:15.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:15.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 10 monclient: dump: 2024-09-13T11:40:15.675 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.675 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.675 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.675 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.675 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.675 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.675 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.675 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.676 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.676 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253bfff640 4 set_mon_vals no callback set 2024-09-13T11:40:15.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7fc288ff9640 1 -- 172.21.15.5:0/1511412036 <== osd.0 v2:172.21.15.5:6804/4278099835 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc2840a57c0 con 0x7fc278040eb0 2024-09-13T11:40:15.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254bef4640 10 monclient: _renew_subs 2024-09-13T11:40:15.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254bef4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254bef4640 1 -- 172.21.15.5:0/865163143 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2544121c00 con 0x7f25300053d0 2024-09-13T11:40:15.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254bef4640 10 monclient: _renew_subs 2024-09-13T11:40:15.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254bef4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254bef4640 1 -- 172.21.15.5:0/865163143 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f25441209c0 con 0x7f25300053d0 2024-09-13T11:40:15.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254bef4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254bef4640 1 -- 172.21.15.5:0/865163143 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f25440a9b60 con 0x7f25300053d0 2024-09-13T11:40:15.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 1 -- 172.21.15.5:0/865163143 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f2540019020 con 0x7f25300053d0 2024-09-13T11:40:15.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 1 --2- 172.21.15.5:0/865163143 >> v2:172.21.15.39:6812/1340838781 conn(0x7f252803b2d0 0x7f252803d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 1 -- 172.21.15.5:0/865163143 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f254004ee30 con 0x7f25300053d0 2024-09-13T11:40:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 1 --2- 172.21.15.5:0/865163143 >> v2:172.21.15.5:6808/1447734001 conn(0x7f2528040de0 0x7f2528043220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 1 -- 172.21.15.5:0/865163143 --> v2:172.21.15.5:6808/1447734001 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25280438b0 con 0x7f2528040de0 2024-09-13T11:40:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 1 -- 172.21.15.5:0/865163143 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f254004f290 con 0x7f25300053d0 2024-09-13T11:40:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f253a7fc640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-13T11:40:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254a491640 1 --2- 172.21.15.5:0/865163143 >> v2:172.21.15.5:6808/1447734001 conn(0x7f2528040de0 0x7f2528043220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254948f640 1 --2- 172.21.15.5:0/865163143 >> v2:172.21.15.39:6812/1340838781 conn(0x7f252803b2d0 0x7f252803d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254a491640 10 monclient: get_auth_request con 0x7f2528040de0 auth_method 0 2024-09-13T11:40:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.627+0000 7f254948f640 10 monclient: get_auth_request con 0x7f252803b2d0 auth_method 0 2024-09-13T11:40:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.631+0000 7f254a491640 1 --2- 172.21.15.5:0/865163143 >> v2:172.21.15.5:6808/1447734001 conn(0x7f2528040de0 0x7f2528043220 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.631+0000 7f254948f640 1 --2- 172.21.15.5:0/865163143 >> v2:172.21.15.39:6812/1340838781 conn(0x7f252803b2d0 0x7f252803d790 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2544120950 tx=0x7f2534008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.631+0000 7f253a7fc640 1 -- 172.21.15.5:0/865163143 <== osd.1 v2:172.21.15.5:6808/1447734001 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f25280438b0 con 0x7f2528040de0 2024-09-13T11:40:15.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.643+0000 7f254bef4640 1 -- 172.21.15.5:0/865163143 --> v2:172.21.15.5:6808/1447734001 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2544074a30 con 0x7f2528040de0 2024-09-13T11:40:15.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f253a7fc640 1 -- 172.21.15.5:0/865163143 <== osd.1 v2:172.21.15.5:6808/1447734001 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2544074a30 con 0x7f2528040de0 2024-09-13T11:40:15.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254bef4640 1 -- 172.21.15.5:0/865163143 >> v2:172.21.15.5:6808/1447734001 conn(0x7f2528040de0 msgr2=0x7f2528043220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254bef4640 1 --2- 172.21.15.5:0/865163143 >> v2:172.21.15.5:6808/1447734001 conn(0x7f2528040de0 0x7f2528043220 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254bef4640 1 -- 172.21.15.5:0/865163143 >> v2:172.21.15.39:6812/1340838781 conn(0x7f252803b2d0 msgr2=0x7f252803d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254bef4640 1 --2- 172.21.15.5:0/865163143 >> v2:172.21.15.39:6812/1340838781 conn(0x7f252803b2d0 0x7f252803d790 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2544120950 tx=0x7f2534008040 comp rx=0 tx=0).stop 2024-09-13T11:40:15.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254bef4640 10 monclient: shutdown 2024-09-13T11:40:15.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254bef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254bef4640 1 -- 172.21.15.5:0/865163143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25300053d0 msgr2=0x7f2544136380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254bef4640 1 --2- 172.21.15.5:0/865163143 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25300053d0 0x7f2544136380 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f254000c520 tx=0x7f254000c9e0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254a491640 1 -- 172.21.15.5:0/865163143 reap_dead start 2024-09-13T11:40:15.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254bef4640 1 -- 172.21.15.5:0/865163143 shutdown_connections 2024-09-13T11:40:15.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254bef4640 1 -- 172.21.15.5:0/865163143 >> 172.21.15.5:0/865163143 conn(0x7f25440a6c70 msgr2=0x7f25440a3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254bef4640 1 -- 172.21.15.5:0/865163143 shutdown_connections 2024-09-13T11:40:15.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.647+0000 7f254bef4640 1 -- 172.21.15.5:0/865163143 wait complete. 2024-09-13T11:40:15.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.655+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 --> v2:172.21.15.5:6804/4278099835 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc28c13c2c0 con 0x7fc278040eb0 2024-09-13T11:40:15.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.655+0000 7fc288ff9640 1 -- 172.21.15.5:0/1511412036 <== osd.0 v2:172.21.15.5:6804/4278099835 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc28c13c2c0 con 0x7fc278040eb0 2024-09-13T11:40:15.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.655+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 >> v2:172.21.15.5:6804/4278099835 conn(0x7fc278040eb0 msgr2=0x7fc2780432f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.655+0000 7fc2922a6640 1 --2- 172.21.15.5:0/1511412036 >> v2:172.21.15.5:6804/4278099835 conn(0x7fc278040eb0 0x7fc2780432f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.659+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 >> v2:172.21.15.39:6812/1340838781 conn(0x7fc27803b320 msgr2=0x7fc27803d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.659+0000 7fc2922a6640 1 --2- 172.21.15.5:0/1511412036 >> v2:172.21.15.39:6812/1340838781 conn(0x7fc27803b320 0x7fc27803d7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc27c009920 tx=0x7fc27c008040 comp rx=0 tx=0).stop 2024-09-13T11:40:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.659+0000 7fc2922a6640 10 monclient: shutdown 2024-09-13T11:40:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.659+0000 7fc2922a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.659+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc28c13cd80 msgr2=0x7fc28c1399d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.659+0000 7fc2922a6640 1 --2- 172.21.15.5:0/1511412036 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc28c13cd80 0x7fc28c1399d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc2840ac000 tx=0x7fc2840195b0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.659+0000 7fc290843640 1 -- 172.21.15.5:0/1511412036 reap_dead start 2024-09-13T11:40:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.659+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 shutdown_connections 2024-09-13T11:40:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.659+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 >> 172.21.15.5:0/1511412036 conn(0x7fc28c0a6c70 msgr2=0x7fc28c13e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.659+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 shutdown_connections 2024-09-13T11:40:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.659+0000 7fc2922a6640 1 -- 172.21.15.5:0/1511412036 wait complete. 2024-09-13T11:40:15.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient: get_monmap_and_config 2024-09-13T11:40:15.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient: monmap: 2024-09-13T11:40:15.687 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.687 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.687 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:15.666116+0000 2024-09-13T11:40:15.687 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:15.666116+0000 2024-09-13T11:40:15.687 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.687 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.688 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:15.688 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:15.688 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:15.688 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:15.688 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:15.688 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 1 Processor -- start 2024-09-13T11:40:15.688 INFO:teuthology.orchestra.run.smithi005.stdout:51539607557 2024-09-13T11:40:15.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 1 -- start start 2024-09-13T11:40:15.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient: init 2024-09-13T11:40:15.689 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T11:40:15.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-13T11:40:15.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0a3980 0x7f795c0a3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient(hunting): picked mon.noname-b con 0x7f795c0a3980 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f795c09ec20 0x7f795c0a42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient(hunting): picked mon.noname-a con 0x7f795c09ec20 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f795c06efd0 0x7f795c137b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient(hunting): picked mon.noname-c con 0x7f795c06efd0 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f795c0a24d0 con 0x7f795c09ec20 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f795c0a47e0 con 0x7f795c06efd0 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f795c0707d0 con 0x7f795c0a3980 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f79614df640 10 monclient(hunting): authenticate will time out at 1019.014666s 2024-09-13T11:40:15.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0a3980 0x7f795c0a3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0a3980 0x7f795c0a3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59144/0 (socket says 172.21.15.5:59144) 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 1 -- 172.21.15.5:0/2091343979 learned_addr learned my addr 172.21.15.5:0/2091343979 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 10 monclient(hunting): get_auth_request con 0x7f795c0a3980 auth_method 0 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795a7fc640 1 --2- 172.21.15.5:0/2091343979 >> v2:172.21.15.5:3300/0 conn(0x7f795c09ec20 0x7f795c0a42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795a7fc640 10 monclient(hunting): get_auth_request con 0x7f795c09ec20 auth_method 0 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795b7fe640 1 --2- 172.21.15.5:0/2091343979 >> v2:172.21.15.39:3300/0 conn(0x7f795c06efd0 0x7f795c137b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795b7fe640 10 monclient(hunting): get_auth_request con 0x7f795c06efd0 auth_method 0 2024-09-13T11:40:15.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient: get_monmap_and_config 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient: monmap: 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:15.669835+0000 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:15.669835+0000 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:15.695 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:15.696 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-09-13T11:40:15.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 1 -- 172.21.15.5:0/2091343979 >> v2:172.21.15.39:3300/0 conn(0x7f795c06efd0 msgr2=0x7f795c137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 1 --2- 172.21.15.5:0/2091343979 >> v2:172.21.15.39:3300/0 conn(0x7f795c06efd0 0x7f795c137b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 1 -- 172.21.15.5:0/2091343979 >> v2:172.21.15.5:3300/0 conn(0x7f795c09ec20 msgr2=0x7f795c0a42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 1 --2- 172.21.15.5:0/2091343979 >> v2:172.21.15.5:3300/0 conn(0x7f795c09ec20 0x7f795c0a42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 1 monclient: found mon.noname-b 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.663+0000 7f795affd640 1 -- 172.21.15.5:0/2091343979 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f795c0704f0 con 0x7f795c0a3980 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f795affd640 1 --2- 172.21.15.5:0/2091343979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0a3980 0x7f795c0a3d60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f795c06ec70 tx=0x7f794c01c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97f8247d68692c34 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 1 -- 172.21.15.5:0/2091343979 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c0b0070 con 0x7f795c0a3980 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 10 monclient: dump: 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 5 monclient: authenticate success, global_id 4232 2024-09-13T11:40:15.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 1 -- 172.21.15.5:0/2091343979 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f794c0ab600 con 0x7f795c0a3980 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 1 -- 172.21.15.5:0/2091343979 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c0aba10 con 0x7f795c0a3980 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 4 set_mon_vals no callback set 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f7959ffb640 10 monclient: dump: 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.699 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.700 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.700 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.700 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 1 Processor -- start 2024-09-13T11:40:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 1 -- start start 2024-09-13T11:40:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient: init 2024-09-13T11:40:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-13T11:40:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa528137930 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa528138420 0x7fa52809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient(hunting): picked mon.noname-a con 0x7fa528138420 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa52813cd80 0x7fa52813f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 10 monclient: shutdown 2024-09-13T11:40:15.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 1 -- 172.21.15.5:0/2091343979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0a3980 msgr2=0x7f795c0a3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 1 --2- 172.21.15.5:0/2091343979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0a3980 0x7f795c0a3d60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f795c06ec70 tx=0x7f794c01c140 comp rx=0 tx=0).stop 2024-09-13T11:40:15.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient(hunting): picked mon.noname-c con 0x7fa52813cd80 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:15.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5280a5a80 con 0x7fa528137930 2024-09-13T11:40:15.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa5280a5fe0 con 0x7fa528138420 2024-09-13T11:40:15.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fa52809c540 con 0x7fa52813cd80 2024-09-13T11:40:15.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52f9ea640 10 monclient(hunting): authenticate will time out at 1019.018525s 2024-09-13T11:40:15.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 1 -- 172.21.15.5:0/2091343979 shutdown_connections 2024-09-13T11:40:15.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 1 --2- 172.21.15.5:0/2091343979 >> v2:172.21.15.39:3300/0 conn(0x7f795c06efd0 0x7f795c137b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 1 --2- 172.21.15.5:0/2091343979 >> v2:172.21.15.5:3300/0 conn(0x7f795c09ec20 0x7f795c0a42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 1 --2- 172.21.15.5:0/2091343979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0a3980 0x7f795c0a3d60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 1 -- 172.21.15.5:0/2091343979 >> 172.21.15.5:0/2091343979 conn(0x7f795c0a85d0 msgr2=0x7f795c0a4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7ff971a96640 10 monclient: get_monmap_and_config 2024-09-13T11:40:15.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7ff971a96640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7ff971a96640 10 monclient: monmap: 2024-09-13T11:40:15.704 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.704 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.704 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:15.671550+0000 2024-09-13T11:40:15.704 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:15.671550+0000 2024-09-13T11:40:15.704 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.704 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.705 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:15.705 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:15.705 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:15.705 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:15.705 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:15.705 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 1 -- 172.21.15.5:0/2091343979 shutdown_connections 2024-09-13T11:40:15.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52df87640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa52813cd80 0x7fa52813f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa528138420 0x7fa52809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 1 -- 172.21.15.5:0/2091343979 wait complete. 2024-09-13T11:40:15.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52df87640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa52813cd80 0x7fa52813f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.5:52808/0 (socket says 172.21.15.5:52808) 2024-09-13T11:40:15.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52d786640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa528137930 0x7fa528137d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.5:34532/0 (socket says 172.21.15.5:34532) 2024-09-13T11:40:15.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52df87640 1 -- 172.21.15.5:0/1346748632 learned_addr learned my addr 172.21.15.5:0/1346748632 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52df87640 10 monclient(hunting): get_auth_request con 0x7fa52813cd80 auth_method 0 2024-09-13T11:40:15.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52df87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52df87640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52df87640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 10 monclient(hunting): get_auth_request con 0x7fa528138420 auth_method 0 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52df87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52df87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52df87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2024-09-13T11:40:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 1 -- 172.21.15.5:0/1346748632 >> v2:172.21.15.39:3300/0 conn(0x7fa52813cd80 msgr2=0x7fa52813f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:15.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 1 --2- 172.21.15.5:0/1346748632 >> v2:172.21.15.39:3300/0 conn(0x7fa52813cd80 0x7fa52813f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 1 -- 172.21.15.5:0/1346748632 >> v1:172.21.15.39:6789/0 conn(0x7fa528137930 legacy=0x7fa528137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 1 monclient: found mon.noname-a 2024-09-13T11:40:15.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7fa52cf85640 1 -- 172.21.15.5:0/1346748632 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52809bef0 con 0x7fa528138420 2024-09-13T11:40:15.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7ff971a96640 1 Processor -- start 2024-09-13T11:40:15.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7ff971a96640 1 -- start start 2024-09-13T11:40:15.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7ff971a96640 10 monclient: init 2024-09-13T11:40:15.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.667+0000 7f79614df640 10 monclient: monmap: 2024-09-13T11:40:15.710 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.710 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:15.711 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 1 Processor -- start 2024-09-13T11:40:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 1 -- start start 2024-09-13T11:40:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient: init 2024-09-13T11:40:15.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-13T11:40:15.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff96c137930 0x7ff96c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 10 monclient(hunting): picked mon.noname-a con 0x7ff96c137930 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:15.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff96c137d10 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:15.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff96c13cd80 0x7ff96c13f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:40:15.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff963fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff96c137d10 0x7ff96c1380f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36806/0 (socket says 172.21.15.5:36806) 2024-09-13T11:40:15.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f795c06efd0 0x7f795c13d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient(hunting): picked mon.noname-b con 0x7f795c06efd0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:15.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0a3980 0x7f795c13b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient(hunting): picked mon.noname-c con 0x7f795c0a3980 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c136cc0 0x7f795c137130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient(hunting): picked mon.noname-a con 0x7f795c136cc0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:15.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 10 monclient(hunting): picked mon.noname-b con 0x7ff96c13cd80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f795c141110 con 0x7f795c136cc0 2024-09-13T11:40:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f795c140f90 con 0x7f795c06efd0 2024-09-13T11:40:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f795c141290 con 0x7f795c0a3980 2024-09-13T11:40:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient(hunting): authenticate will time out at 1019.020669s 2024-09-13T11:40:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795b7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c136cc0 0x7f795c137130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795b7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c136cc0 0x7f795c137130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57282/0 (socket says 172.21.15.5:57282) 2024-09-13T11:40:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795b7fe640 1 -- 172.21.15.5:0/3353783547 learned_addr learned my addr 172.21.15.5:0/3353783547 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795b7fe640 10 monclient(hunting): get_auth_request con 0x7f795c136cc0 auth_method 0 2024-09-13T11:40:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 1 --2- 172.21.15.5:0/3353783547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0a3980 0x7f795c13b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 10 monclient(hunting): get_auth_request con 0x7f795c0a3980 auth_method 0 2024-09-13T11:40:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff963fff640 1 -- 172.21.15.5:0/500847767 learned_addr learned my addr 172.21.15.5:0/500847767 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 1 --2- 172.21.15.5:0/500847767 >> v2:172.21.15.5:3300/0 conn(0x7ff96c137930 0x7ff96c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96bfff640 1 --2- 172.21.15.5:0/500847767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff96c13cd80 0x7ff96c13f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795affd640 1 --2- 172.21.15.5:0/3353783547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f795c06efd0 0x7f795c13d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795affd640 10 monclient(hunting): get_auth_request con 0x7f795c06efd0 auth_method 0 2024-09-13T11:40:15.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-09-13T11:40:15.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 1 -- 172.21.15.5:0/3353783547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f795c06efd0 msgr2=0x7f795c13d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 1 --2- 172.21.15.5:0/3353783547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f795c06efd0 0x7f795c13d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 1 -- 172.21.15.5:0/500847767 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff96c0a5a80 con 0x7ff96c137d10 2024-09-13T11:40:15.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 1 -- 172.21.15.5:0/3353783547 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c136cc0 msgr2=0x7f795c137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 1 --2- 172.21.15.5:0/3353783547 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c136cc0 0x7f795c137130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 1 monclient: found mon.noname-c 2024-09-13T11:40:15.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 1 -- 172.21.15.5:0/3353783547 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f795c13c690 con 0x7f795c0a3980 2024-09-13T11:40:15.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795affd640 1 --2- 172.21.15.5:0/3353783547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f795c06efd0 0x7f795c13d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:40:15.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795b7fe640 1 --2- 172.21.15.5:0/3353783547 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f795c136cc0 0x7f795c137130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 1 -- 172.21.15.5:0/500847767 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff96c0a5fe0 con 0x7ff96c137930 2024-09-13T11:40:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 1 -- 172.21.15.5:0/500847767 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7ff96c09c540 con 0x7ff96c13cd80 2024-09-13T11:40:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff971a96640 10 monclient(hunting): authenticate will time out at 1019.021619s 2024-09-13T11:40:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f795a7fc640 1 --2- 172.21.15.5:0/3353783547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0a3980 0x7f795c13b0e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f795000e8a0 tx=0x7f795000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 10 monclient(hunting): get_auth_request con 0x7ff96c137930 auth_method 0 2024-09-13T11:40:15.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96affd640 1 -- 172.21.15.5:0/500847767 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1373046408 0 0) 0x7ff96c0a5a80 con 0x7ff96c137d10 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96affd640 10 monclient(hunting): _negotiate 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96affd640 10 monclient(hunting): my global_id is 4278 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96affd640 1 -- 172.21.15.5:0/500847767 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff958003530 con 0x7ff96c137d10 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96bfff640 10 monclient(hunting): get_auth_request con 0x7ff96c13cd80 auth_method 0 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 1 -- 172.21.15.5:0/3353783547 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950017070 con 0x7f795c0a3980 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 10 monclient: dump: 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 1 -- 172.21.15.5:0/3353783547 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f795000c940 con 0x7f795c0a3980 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 1 -- 172.21.15.5:0/3353783547 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795000ccf0 con 0x7f795c0a3980 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79437fe640 10 monclient: dump: 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.727 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.727 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.727 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.727 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 5 monclient: authenticate success, global_id 4235 2024-09-13T11:40:15.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient: _renew_subs 2024-09-13T11:40:15.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 1 -- 172.21.15.5:0/3353783547 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f795c13b620 con 0x7f795c0a3980 2024-09-13T11:40:15.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79597fa640 4 set_mon_vals no callback set 2024-09-13T11:40:15.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient: _renew_subs 2024-09-13T11:40:15.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 1 -- 172.21.15.5:0/3353783547 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f795c1cf6d0 con 0x7f795c0a3980 2024-09-13T11:40:15.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa52cf85640 1 --2- 172.21.15.5:0/1346748632 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa528138420 0x7fa52809cc90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa528138800 tx=0x7fa51801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d92ef95fd3a3d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7f79614df640 1 -- 172.21.15.5:0/3353783547 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f795c0a14b0 con 0x7f795c0a3980 2024-09-13T11:40:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 1 -- 172.21.15.5:0/1346748632 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5180b0070 con 0x7fa528138420 2024-09-13T11:40:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 10 monclient: dump: 2024-09-13T11:40:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 1 -- 172.21.15.5:0/1346748632 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5180ab600 con 0x7fa528138420 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 1 -- 172.21.15.5:0/1346748632 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5180ab9d0 con 0x7fa528138420 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa51ffff640 10 monclient: dump: 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.730 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.731 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.731 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.731 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.731 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.731 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.731 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.731 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa52f9ea640 5 monclient: authenticate success, global_id 4272 2024-09-13T11:40:15.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa52f9ea640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:15.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7fa52f9ea640 4 set_mon_vals no callback set 2024-09-13T11:40:15.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2024-09-13T11:40:15.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 1 -- 172.21.15.5:0/500847767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff96c13cd80 msgr2=0x7ff96c13f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:15.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 1 --2- 172.21.15.5:0/500847767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff96c13cd80 0x7ff96c13f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 1 -- 172.21.15.5:0/500847767 >> v1:172.21.15.5:6789/0 conn(0x7ff96c137d10 legacy=0x7ff96c1380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 1 monclient: found mon.noname-a 2024-09-13T11:40:15.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.671+0000 7ff96b7fe640 1 -- 172.21.15.5:0/500847767 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96c09bef0 con 0x7ff96c137930 2024-09-13T11:40:15.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96bfff640 1 --2- 172.21.15.5:0/500847767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff96c13cd80 0x7ff96c13f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:40:15.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 10 monclient: get_monmap_and_config 2024-09-13T11:40:15.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient: shutdown 2024-09-13T11:40:15.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 -- 172.21.15.5:0/1346748632 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa528138420 msgr2=0x7fa52809cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 --2- 172.21.15.5:0/1346748632 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa528138420 0x7fa52809cc90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa528138800 tx=0x7fa51801c340 comp rx=0 tx=0).stop 2024-09-13T11:40:15.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96b7fe640 1 --2- 172.21.15.5:0/500847767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff96c137930 0x7ff96c09cc90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff95c0097c0 tx=0x7ff95c01c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f750f303e9e0081 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 10 monclient: monmap: 2024-09-13T11:40:15.734 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.734 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.734 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:15.677346+0000 2024-09-13T11:40:15.734 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:15.677346+0000 2024-09-13T11:40:15.734 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.734 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.734 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:15.735 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:15.735 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:15.735 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:15.735 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:15.735 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 -- 172.21.15.5:0/1346748632 shutdown_connections 2024-09-13T11:40:15.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 --2- 172.21.15.5:0/1346748632 >> v2:172.21.15.39:3300/0 conn(0x7fa52813cd80 0x7fa52813f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 --2- 172.21.15.5:0/1346748632 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa528138420 0x7fa52809cc90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 -- 172.21.15.5:0/1346748632 >> 172.21.15.5:0/1346748632 conn(0x7fa5280a6c70 msgr2=0x7fa5280a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f79437fe640 1 -- 172.21.15.5:0/3353783547 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f795001a070 con 0x7f795c0a3980 2024-09-13T11:40:15.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f79437fe640 1 --2- 172.21.15.5:0/3353783547 >> v2:172.21.15.39:6812/1340838781 conn(0x7f794803b2f0 0x7f794803d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f79437fe640 1 -- 172.21.15.5:0/3353783547 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f795004f080 con 0x7f795c0a3980 2024-09-13T11:40:15.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f79437fe640 1 --2- 172.21.15.5:0/3353783547 >> v2:172.21.15.39:6804/3188120659 conn(0x7f7948040e00 0x7f7948043240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f79437fe640 1 -- 172.21.15.5:0/3353783547 --> v2:172.21.15.39:6804/3188120659 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7950022e50 con 0x7f7948040e00 2024-09-13T11:40:15.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f79437fe640 1 -- 172.21.15.5:0/3353783547 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f795004f330 con 0x7f795c0a3980 2024-09-13T11:40:15.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f79437fe640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-13T11:40:15.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 -- 172.21.15.5:0/1346748632 shutdown_connections 2024-09-13T11:40:15.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 -- 172.21.15.5:0/1346748632 wait complete. 2024-09-13T11:40:15.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f795affd640 1 --2- 172.21.15.5:0/3353783547 >> v2:172.21.15.39:6812/1340838781 conn(0x7f794803b2f0 0x7f794803d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 1 -- 172.21.15.5:0/500847767 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0b0070 con 0x7ff96c137930 2024-09-13T11:40:15.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 10 monclient: dump: 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 1 -- 172.21.15.5:0/500847767 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff95c0ab400 con 0x7ff96c137930 2024-09-13T11:40:15.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 1 -- 172.21.15.5:0/500847767 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff954001700 con 0x7ff96c137930 2024-09-13T11:40:15.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:15.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f795affd640 10 monclient: get_auth_request con 0x7f794803b2f0 auth_method 0 2024-09-13T11:40:15.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff96affd640 10 monclient: dump: 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 5 monclient: authenticate success, global_id 4281 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:15.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 4 set_mon_vals no callback set 2024-09-13T11:40:15.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f795b7fe640 1 --2- 172.21.15.5:0/3353783547 >> v2:172.21.15.39:6804/3188120659 conn(0x7f7948040e00 0x7f7948043240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f795b7fe640 10 monclient: get_auth_request con 0x7f7948040e00 auth_method 0 2024-09-13T11:40:15.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 1 Processor -- start 2024-09-13T11:40:15.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 1 -- start start 2024-09-13T11:40:15.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 10 monclient: init 2024-09-13T11:40:15.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient: monmap: 2024-09-13T11:40:15.742 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.742 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:15.743 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:15.743 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:15.743 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.743 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.743 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:15.743 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:15.743 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:15.743 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f795affd640 1 --2- 172.21.15.5:0/3353783547 >> v2:172.21.15.39:6812/1340838781 conn(0x7f794803b2f0 0x7f794803d7b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f794c0ad000 tx=0x7f794c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 Processor -- start 2024-09-13T11:40:15.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 -- start start 2024-09-13T11:40:15.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient: init 2024-09-13T11:40:15.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7f795b7fe640 1 --2- 172.21.15.5:0/3353783547 >> v2:172.21.15.39:6804/3188120659 conn(0x7f7948040e00 0x7f7948043240 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 10 monclient: shutdown 2024-09-13T11:40:15.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 1 -- 172.21.15.5:0/500847767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff96c137930 msgr2=0x7ff96c09cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 1 --2- 172.21.15.5:0/500847767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff96c137930 0x7ff96c09cc90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff95c0097c0 tx=0x7ff95c01c1c0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-13T11:40:15.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe400137930 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:15.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe400138420 0x7fe40009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 10 monclient(hunting): picked mon.noname-c con 0x7fe400138420 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:15.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe40013cd80 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:15.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3feffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fe400137930 0x7fe400137d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36822/0 (socket says 172.21.15.5:36822) 2024-09-13T11:40:15.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3feffd640 1 -- 172.21.15.5:0/2215661519 learned_addr learned my addr 172.21.15.5:0/2215661519 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:15.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 1 -- 172.21.15.5:0/500847767 shutdown_connections 2024-09-13T11:40:15.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 1 --2- 172.21.15.5:0/500847767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff96c13cd80 0x7ff96c13f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 1 --2- 172.21.15.5:0/500847767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff96c137930 0x7ff96c09cc90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 1 -- 172.21.15.5:0/500847767 >> 172.21.15.5:0/500847767 conn(0x7ff96c0a6c70 msgr2=0x7ff96c0a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:40:15.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa528137930 0x7fa528136820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient(hunting): picked mon.noname-a con 0x7fa528137930 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:15.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa528138420 0x7fa528139a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient(hunting): picked mon.noname-c con 0x7fa528138420 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa52813cd80 0x7fa528139f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient(hunting): picked mon.noname-b con 0x7fa52813cd80 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:15.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa5281422f0 con 0x7fa528137930 2024-09-13T11:40:15.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa528142170 con 0x7fa52813cd80 2024-09-13T11:40:15.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa528142470 con 0x7fa528138420 2024-09-13T11:40:15.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient(hunting): authenticate will time out at 1019.026459s 2024-09-13T11:40:15.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 1 -- 172.21.15.5:0/500847767 shutdown_connections 2024-09-13T11:40:15.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa528138420 0x7fa528139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa528138420 0x7fa528139a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59166/0 (socket says 172.21.15.5:59166) 2024-09-13T11:40:15.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 1 -- 172.21.15.5:0/3933300436 learned_addr learned my addr 172.21.15.5:0/3933300436 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 10 monclient(hunting): get_auth_request con 0x7fa528138420 auth_method 0 2024-09-13T11:40:15.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 1 -- 172.21.15.5:0/2215661519 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4000a5a80 con 0x7fe400137930 2024-09-13T11:40:15.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 1 -- 172.21.15.5:0/2215661519 --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe40009bef0 con 0x7fe40013cd80 2024-09-13T11:40:15.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 1 -- 172.21.15.5:0/500847767 wait complete. 2024-09-13T11:40:15.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 1 -- 172.21.15.5:0/2215661519 --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fe4000a5fe0 con 0x7fe400138420 2024-09-13T11:40:15.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 1 --2- 172.21.15.5:0/2215661519 >> v2:172.21.15.39:3300/0 conn(0x7fe400138420 0x7fe40009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe405633640 10 monclient(hunting): authenticate will time out at 1019.026886s 2024-09-13T11:40:15.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52df87640 1 --2- 172.21.15.5:0/3933300436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa52813cd80 0x7fa528139f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 10 monclient(hunting): get_auth_request con 0x7fe400138420 auth_method 0 2024-09-13T11:40:15.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52d786640 1 --2- 172.21.15.5:0/3933300436 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa528137930 0x7fa528136820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52d786640 10 monclient(hunting): get_auth_request con 0x7fa528137930 auth_method 0 2024-09-13T11:40:15.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52d786640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52d786640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52d786640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 10 monclient(hunting): handle_auth_done global_id 4241 payload 824 2024-09-13T11:40:15.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 1 -- 172.21.15.5:0/3933300436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa52813cd80 msgr2=0x7fa528139f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 1 --2- 172.21.15.5:0/3933300436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa52813cd80 0x7fa528139f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 1 -- 172.21.15.5:0/3933300436 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa528137930 msgr2=0x7fa528136820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 1 --2- 172.21.15.5:0/3933300436 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa528137930 0x7fa528136820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 1 monclient: found mon.noname-c 2024-09-13T11:40:15.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 1 -- 172.21.15.5:0/3933300436 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5280a1360 con 0x7fa528138420 2024-09-13T11:40:15.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fdffb640 1 -- 172.21.15.5:0/2215661519 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1517490517 0 0) 0x7fe4000a5a80 con 0x7fe400137930 2024-09-13T11:40:15.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fdffb640 10 monclient(hunting): _negotiate 2024-09-13T11:40:15.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fdffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fdffb640 10 monclient(hunting): my global_id is 4284 2024-09-13T11:40:15.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fdffb640 1 -- 172.21.15.5:0/2215661519 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3e8003530 con 0x7fe400137930 2024-09-13T11:40:15.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52cf85640 1 --2- 172.21.15.5:0/3933300436 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa528138420 0x7fa528139a40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa518018f50 tx=0x7fa5180190e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 10 monclient(hunting): handle_auth_done global_id 4285 payload 274 2024-09-13T11:40:15.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 1 -- 172.21.15.5:0/2215661519 >> v1:172.21.15.39:6789/0 conn(0x7fe40013cd80 legacy=0x7fe40013f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 1 -- 172.21.15.5:0/2215661519 >> v1:172.21.15.5:6789/0 conn(0x7fe400137930 legacy=0x7fe400137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 1 monclient: found mon.noname-c 2024-09-13T11:40:15.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.39:3300/0 2024-09-13T11:40:15.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fe3fe7fc640 1 -- 172.21.15.5:0/2215661519 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40009c1d0 con 0x7fe400138420 2024-09-13T11:40:15.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 1 -- 172.21.15.5:0/3933300436 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5180b8020 con 0x7fa528138420 2024-09-13T11:40:15.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:15.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 10 monclient: dump: 2024-09-13T11:40:15.757 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.757 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.757 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.757 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.758 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 1 -- 172.21.15.5:0/3933300436 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5180b4600 con 0x7fa528138420 2024-09-13T11:40:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 1 -- 172.21.15.5:0/3933300436 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5180b0040 con 0x7fa528138420 2024-09-13T11:40:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51dffb640 10 monclient: dump: 2024-09-13T11:40:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.760 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.760 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7ff971a96640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient: monmap: 2024-09-13T11:40:15.760 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.760 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:15.760 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:15.760 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:15.761 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.761 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.761 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:15.761 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:15.761 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:15.761 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7f79437fe640 1 -- 172.21.15.5:0/3353783547 <== osd.3 v2:172.21.15.39:6804/3188120659 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7950022e50 con 0x7f7948040e00 2024-09-13T11:40:15.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 1 Processor -- start 2024-09-13T11:40:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 1 -- start start 2024-09-13T11:40:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 5 monclient: authenticate success, global_id 4241 2024-09-13T11:40:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient: _renew_subs 2024-09-13T11:40:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 -- 172.21.15.5:0/3933300436 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa528120fd0 con 0x7fa528138420 2024-09-13T11:40:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa51f7fe640 4 set_mon_vals no callback set 2024-09-13T11:40:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient: _renew_subs 2024-09-13T11:40:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.675+0000 7fa52f9ea640 1 -- 172.21.15.5:0/3933300436 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5280a9490 con 0x7fa528138420 2024-09-13T11:40:15.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient: init 2024-09-13T11:40:15.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fa52f9ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fa52f9ea640 1 -- 172.21.15.5:0/3933300436 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa52809cc90 con 0x7fa528138420 2024-09-13T11:40:15.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff96c137930 0x7ff96c13bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient(hunting): picked mon.noname-b con 0x7ff96c137930 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff96c137d10 0x7ff96c1cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient(hunting): picked mon.noname-c con 0x7ff96c137d10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff96c13cd80 0x7ff96c1399b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff963fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff96c137d10 0x7ff96c1cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff963fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff96c137d10 0x7ff96c1cfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59176/0 (socket says 172.21.15.5:59176) 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96b7fe640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff96c137930 0x7ff96c13bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient(hunting): picked mon.noname-a con 0x7ff96c13cd80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff963fff640 1 -- 172.21.15.5:0/1238392242 learned_addr learned my addr 172.21.15.5:0/1238392242 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff96c142340 con 0x7ff96c13cd80 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff96c1421c0 con 0x7ff96c137930 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7ff96c1424c0 con 0x7ff96c137d10 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient(hunting): authenticate will time out at 1019.030022s 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff963fff640 10 monclient(hunting): get_auth_request con 0x7ff96c137d10 auth_method 0 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff963fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff963fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff963fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96b7fe640 10 monclient(hunting): get_auth_request con 0x7ff96c137930 auth_method 0 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 1 --2- 172.21.15.5:0/1238392242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff96c13cd80 0x7ff96c1399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fe7fc640 1 --2- 172.21.15.5:0/2215661519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400138420 0x7fe40009cc90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40097c0 tx=0x7fe3f401c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d52addc88dfe008 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 10 monclient(hunting): get_auth_request con 0x7ff96c13cd80 auth_method 0 2024-09-13T11:40:15.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff963fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff963fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff963fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 1 -- 172.21.15.5:0/1238392242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff96c137d10 msgr2=0x7ff96c1cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 1 --2- 172.21.15.5:0/1238392242 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff96c137d10 0x7ff96c1cfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 1 -- 172.21.15.5:0/1238392242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff96c137930 msgr2=0x7ff96c13bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 1 --2- 172.21.15.5:0/1238392242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff96c137930 0x7ff96c13bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 1 monclient: found mon.noname-a 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 1 -- 172.21.15.5:0/1238392242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96c121170 con 0x7ff96c13cd80 2024-09-13T11:40:15.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 1 -- 172.21.15.5:0/2215661519 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f40b0070 con 0x7fe400138420 2024-09-13T11:40:15.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:15.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 10 monclient: dump: 2024-09-13T11:40:15.770 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.770 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.770 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.770 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.770 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 1 -- 172.21.15.5:0/2215661519 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f40ab600 con 0x7fe400138420 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 1 -- 172.21.15.5:0/2215661519 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f40ab9d0 con 0x7fe400138420 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:40:15.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe3fdffb640 10 monclient: dump: 2024-09-13T11:40:15.772 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.772 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.772 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.772 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.772 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.772 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.772 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.772 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.772 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.772 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96bfff640 1 --2- 172.21.15.5:0/1238392242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff96c13cd80 0x7ff96c1399b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff964009f90 tx=0x7ff96400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe405633640 5 monclient: authenticate success, global_id 4285 2024-09-13T11:40:15.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe405633640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:15.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe405633640 4 set_mon_vals no callback set 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 1 -- 172.21.15.5:0/1238392242 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff964003ba0 con 0x7ff96c13cd80 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 10 monclient: dump: 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.774 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 1 -- 172.21.15.5:0/1238392242 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff964003d40 con 0x7ff96c13cd80 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 5 monclient: authenticate success, global_id 4290 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient: _renew_subs 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff96c09db50 con 0x7ff96c13cd80 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 1 -- 172.21.15.5:0/1238392242 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff964004220 con 0x7ff96c13cd80 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:15.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff968ff9640 10 monclient: dump: 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient: _renew_subs 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff96a7fc640 4 set_mon_vals no callback set 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff96c120e90 con 0x7ff96c13cd80 2024-09-13T11:40:15.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe405633640 10 monclient: shutdown 2024-09-13T11:40:15.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe405633640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe405633640 1 -- 172.21.15.5:0/2215661519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400138420 msgr2=0x7fe40009cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe405633640 1 --2- 172.21.15.5:0/2215661519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400138420 0x7fe40009cc90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40097c0 tx=0x7fe3f401c380 comp rx=0 tx=0).stop 2024-09-13T11:40:15.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe405633640 1 -- 172.21.15.5:0/2215661519 shutdown_connections 2024-09-13T11:40:15.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe405633640 1 --2- 172.21.15.5:0/2215661519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400138420 0x7fe40009cc90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.679+0000 7fe405633640 1 -- 172.21.15.5:0/2215661519 >> 172.21.15.5:0/2215661519 conn(0x7fe4000a6c70 msgr2=0x7fe4000a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 1 -- 172.21.15.5:0/2215661519 shutdown_connections 2024-09-13T11:40:15.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 1 -- 172.21.15.5:0/2215661519 wait complete. 2024-09-13T11:40:15.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff971a96640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff96c0a7d60 con 0x7ff96c13cd80 2024-09-13T11:40:15.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff968ff9640 1 -- 172.21.15.5:0/1238392242 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7ff9640043c0 con 0x7ff96c13cd80 2024-09-13T11:40:15.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff968ff9640 1 --2- 172.21.15.5:0/1238392242 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff94c03b2a0 0x7ff94c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff968ff9640 1 -- 172.21.15.5:0/1238392242 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7ff96404f240 con 0x7ff96c13cd80 2024-09-13T11:40:15.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fa51dffb640 1 -- 172.21.15.5:0/3933300436 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fa5180bb020 con 0x7fa528138420 2024-09-13T11:40:15.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff968ff9640 1 --2- 172.21.15.5:0/1238392242 >> v2:172.21.15.39:6808/1185247880 conn(0x7ff94c040db0 0x7ff94c0431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff968ff9640 1 -- 172.21.15.5:0/1238392242 --> v2:172.21.15.39:6808/1185247880 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff94c043880 con 0x7ff94c040db0 2024-09-13T11:40:15.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff968ff9640 1 -- 172.21.15.5:0/1238392242 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff964056050 con 0x7ff96c13cd80 2024-09-13T11:40:15.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff968ff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-13T11:40:15.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fa51dffb640 1 --2- 172.21.15.5:0/3933300436 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa50c03b2d0 0x7fa50c03d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fa51dffb640 1 -- 172.21.15.5:0/3933300436 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fa5180e9670 con 0x7fa528138420 2024-09-13T11:40:15.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fa51dffb640 1 --2- 172.21.15.5:0/3933300436 >> v2:172.21.15.5:6800/1907498696 conn(0x7fa50c040de0 0x7fa50c043220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fa51dffb640 1 -- 172.21.15.5:0/3933300436 --> v2:172.21.15.5:6800/1907498696 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa518006800 con 0x7fa50c040de0 2024-09-13T11:40:15.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fa51dffb640 1 -- 172.21.15.5:0/3933300436 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa5180e99e0 con 0x7fa528138420 2024-09-13T11:40:15.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fa51dffb640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-13T11:40:15.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff96b7fe640 1 --2- 172.21.15.5:0/1238392242 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff94c03b2a0 0x7ff94c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fa52d786640 1 --2- 172.21.15.5:0/3933300436 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa50c03b2d0 0x7fa50c03d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fa52d786640 10 monclient: get_auth_request con 0x7fa50c03b2d0 auth_method 0 2024-09-13T11:40:15.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff96b7fe640 10 monclient: get_auth_request con 0x7ff94c03b2a0 auth_method 0 2024-09-13T11:40:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient: monmap: 2024-09-13T11:40:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:15.783 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 1 Processor -- start 2024-09-13T11:40:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fa52d786640 1 --2- 172.21.15.5:0/3933300436 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa50c03b2d0 0x7fa50c03d790 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa5200012f0 tx=0x7fa520005b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff963fff640 1 --2- 172.21.15.5:0/1238392242 >> v2:172.21.15.39:6808/1185247880 conn(0x7ff94c040db0 0x7ff94c0431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 1 -- start start 2024-09-13T11:40:15.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient: init 2024-09-13T11:40:15.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff96b7fe640 1 --2- 172.21.15.5:0/1238392242 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff94c03b2a0 0x7ff94c03d760 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff95c0ad000 tx=0x7ff95c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff963fff640 10 monclient: get_auth_request con 0x7ff94c040db0 auth_method 0 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400137930 0x7fe400138d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient(hunting): picked mon.noname-b con 0x7fe400137930 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe400138420 0x7fe4001392a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient(hunting): picked mon.noname-a con 0x7fe400138420 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe40013cd80 0x7fe4001397e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient(hunting): picked mon.noname-c con 0x7fe40013cd80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe400143850 con 0x7fe400138420 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe4001436d0 con 0x7fe400137930 2024-09-13T11:40:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fe4001439d0 con 0x7fe40013cd80 2024-09-13T11:40:15.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient(hunting): authenticate will time out at 1019.034606s 2024-09-13T11:40:15.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7ff963fff640 1 --2- 172.21.15.5:0/1238392242 >> v2:172.21.15.39:6808/1185247880 conn(0x7ff94c040db0 0x7ff94c0431f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe40013cd80 0x7fe4001397e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe40013cd80 0x7fe4001397e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59178/0 (socket says 172.21.15.5:59178) 2024-09-13T11:40:15.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 1 -- 172.21.15.5:0/3543121427 learned_addr learned my addr 172.21.15.5:0/3543121427 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.788 INFO:teuthology.orchestra.run.smithi005.stdout:47244640261 2024-09-13T11:40:15.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 10 monclient(hunting): get_auth_request con 0x7fe40013cd80 auth_method 0 2024-09-13T11:40:15.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.789 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T11:40:15.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3fe7fc640 1 --2- 172.21.15.5:0/3543121427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe400138420 0x7fe4001392a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3fe7fc640 10 monclient(hunting): get_auth_request con 0x7fe400138420 auth_method 0 2024-09-13T11:40:15.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-09-13T11:40:15.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 1 -- 172.21.15.5:0/3543121427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400137930 msgr2=0x7fe400138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 1 --2- 172.21.15.5:0/3543121427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe400137930 0x7fe400138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 1 -- 172.21.15.5:0/3543121427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe400138420 msgr2=0x7fe4001392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 1 --2- 172.21.15.5:0/3543121427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe400138420 0x7fe4001392a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 1 monclient: found mon.noname-c 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 1 -- 172.21.15.5:0/3543121427 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40013c210 con 0x7fe40013cd80 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3ff7fe640 1 --2- 172.21.15.5:0/3543121427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe40013cd80 0x7fe4001397e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800c7c0 tx=0x7fe3f800cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 1 -- 172.21.15.5:0/3543121427 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8003b80 con 0x7fe40013cd80 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 10 monclient: dump: 2024-09-13T11:40:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 1 -- 172.21.15.5:0/3543121427 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f8003d20 con 0x7fe40013cd80 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 5 monclient: authenticate success, global_id 4247 2024-09-13T11:40:15.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient: _renew_subs 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 1 -- 172.21.15.5:0/3543121427 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe40013c4f0 con 0x7fe40013cd80 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 1 -- 172.21.15.5:0/3543121427 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f800aa90 con 0x7fe40013cd80 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3e77fe640 10 monclient: dump: 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.795 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.796 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.796 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.796 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe3fd7fa640 4 set_mon_vals no callback set 2024-09-13T11:40:15.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient: _renew_subs 2024-09-13T11:40:15.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.683+0000 7fe405633640 1 -- 172.21.15.5:0/3543121427 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4000a1360 con 0x7fe40013cd80 2024-09-13T11:40:15.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.687+0000 7fa52df87640 1 --2- 172.21.15.5:0/3933300436 >> v2:172.21.15.5:6800/1907498696 conn(0x7fa50c040de0 0x7fa50c043220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.687+0000 7fa52df87640 10 monclient: get_auth_request con 0x7fa50c040de0 auth_method 0 2024-09-13T11:40:15.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.687+0000 7ff968ff9640 1 -- 172.21.15.5:0/1238392242 <== osd.5 v2:172.21.15.39:6808/1185247880 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff94c043880 con 0x7ff94c040db0 2024-09-13T11:40:15.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.687+0000 7fe405633640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.687+0000 7fe405633640 1 -- 172.21.15.5:0/3543121427 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe4001405c0 con 0x7fe40013cd80 2024-09-13T11:40:15.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.687+0000 7fa52df87640 1 --2- 172.21.15.5:0/3933300436 >> v2:172.21.15.5:6800/1907498696 conn(0x7fa50c040de0 0x7fa50c043220 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.687+0000 7fa51dffb640 1 -- 172.21.15.5:0/3933300436 <== osd.2 v2:172.21.15.5:6800/1907498696 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa518006800 con 0x7fa50c040de0 2024-09-13T11:40:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.691+0000 7fe3e77fe640 1 -- 172.21.15.5:0/3543121427 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fe3f800ac30 con 0x7fe40013cd80 2024-09-13T11:40:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.691+0000 7fe3e77fe640 1 --2- 172.21.15.5:0/3543121427 >> v2:172.21.15.39:6812/1340838781 conn(0x7fe3ec03b2d0 0x7fe3ec03d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.691+0000 7fe3e77fe640 1 -- 172.21.15.5:0/3543121427 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fe3f804eb10 con 0x7fe40013cd80 2024-09-13T11:40:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.691+0000 7fe3e77fe640 1 --2- 172.21.15.5:0/3543121427 >> v2:172.21.15.39:6800/4264535364 conn(0x7fe3ec040de0 0x7fe3ec043220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.691+0000 7fe3e77fe640 1 -- 172.21.15.5:0/3543121427 --> v2:172.21.15.39:6800/4264535364 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3ec0438b0 con 0x7fe3ec040de0 2024-09-13T11:40:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.691+0000 7fe3feffd640 1 --2- 172.21.15.5:0/3543121427 >> v2:172.21.15.39:6812/1340838781 conn(0x7fe3ec03b2d0 0x7fe3ec03d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.691+0000 7fe3fe7fc640 1 --2- 172.21.15.5:0/3543121427 >> v2:172.21.15.39:6800/4264535364 conn(0x7fe3ec040de0 0x7fe3ec043220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.691+0000 7fe3feffd640 10 monclient: get_auth_request con 0x7fe3ec03b2d0 auth_method 0 2024-09-13T11:40:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.691+0000 7fe3fe7fc640 10 monclient: get_auth_request con 0x7fe3ec040de0 auth_method 0 2024-09-13T11:40:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.691+0000 7fe3fe7fc640 1 --2- 172.21.15.5:0/3543121427 >> v2:172.21.15.39:6800/4264535364 conn(0x7fe3ec040de0 0x7fe3ec043220 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.691+0000 7fe3feffd640 1 --2- 172.21.15.5:0/3543121427 >> v2:172.21.15.39:6812/1340838781 conn(0x7fe3ec03b2d0 0x7fe3ec03d790 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0007fd0 tx=0x7fe3f0007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.691+0000 7fe3e77fe640 1 -- 172.21.15.5:0/3543121427 <== osd.4 v2:172.21.15.39:6800/4264535364 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3ec0438b0 con 0x7fe3ec040de0 2024-09-13T11:40:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.695+0000 7fe3e77fe640 1 -- 172.21.15.5:0/3543121427 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe3f80052b0 con 0x7fe40013cd80 2024-09-13T11:40:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.695+0000 7fe3e77fe640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-13T11:40:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.699+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 --> v2:172.21.15.39:6808/1185247880 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff96c0a1140 con 0x7ff94c040db0 2024-09-13T11:40:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.699+0000 7ff968ff9640 1 -- 172.21.15.5:0/1238392242 <== osd.5 v2:172.21.15.39:6808/1185247880 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff96c0a1140 con 0x7ff94c040db0 2024-09-13T11:40:15.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 >> v2:172.21.15.39:6808/1185247880 conn(0x7ff94c040db0 msgr2=0x7ff94c0431f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff971a96640 1 --2- 172.21.15.5:0/1238392242 >> v2:172.21.15.39:6808/1185247880 conn(0x7ff94c040db0 0x7ff94c0431f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff94c03b2a0 msgr2=0x7ff94c03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff971a96640 1 --2- 172.21.15.5:0/1238392242 >> v2:172.21.15.39:6812/1340838781 conn(0x7ff94c03b2a0 0x7ff94c03d760 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff95c0ad000 tx=0x7ff95c008040 comp rx=0 tx=0).stop 2024-09-13T11:40:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff971a96640 10 monclient: shutdown 2024-09-13T11:40:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff971a96640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff96c13cd80 msgr2=0x7ff96c1399b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff971a96640 1 --2- 172.21.15.5:0/1238392242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff96c13cd80 0x7ff96c1399b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff964009f90 tx=0x7ff96400d560 comp rx=0 tx=0).stop 2024-09-13T11:40:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff96bfff640 1 -- 172.21.15.5:0/1238392242 reap_dead start 2024-09-13T11:40:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 shutdown_connections 2024-09-13T11:40:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 >> 172.21.15.5:0/1238392242 conn(0x7ff96c0a6c70 msgr2=0x7ff96c13dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 shutdown_connections 2024-09-13T11:40:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7ff971a96640 1 -- 172.21.15.5:0/1238392242 wait complete. 2024-09-13T11:40:15.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 1 -- 172.21.15.5:0/3353783547 --> v2:172.21.15.39:6804/3188120659 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f795c09f160 con 0x7f7948040e00 2024-09-13T11:40:15.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79437fe640 1 -- 172.21.15.5:0/3353783547 <== osd.3 v2:172.21.15.39:6804/3188120659 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f795c09f160 con 0x7f7948040e00 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 1 -- 172.21.15.5:0/3353783547 >> v2:172.21.15.39:6804/3188120659 conn(0x7f7948040e00 msgr2=0x7f7948043240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 1 --2- 172.21.15.5:0/3353783547 >> v2:172.21.15.39:6804/3188120659 conn(0x7f7948040e00 0x7f7948043240 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 1 -- 172.21.15.5:0/3353783547 >> v2:172.21.15.39:6812/1340838781 conn(0x7f794803b2f0 msgr2=0x7f794803d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 1 --2- 172.21.15.5:0/3353783547 >> v2:172.21.15.39:6812/1340838781 conn(0x7f794803b2f0 0x7f794803d7b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f794c0ad000 tx=0x7f794c0a6000 comp rx=0 tx=0).stop 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 10 monclient: shutdown 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 1 -- 172.21.15.5:0/3353783547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0a3980 msgr2=0x7f795c13b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 1 --2- 172.21.15.5:0/3353783547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f795c0a3980 0x7f795c13b0e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f795000e8a0 tx=0x7f795000ed60 comp rx=0 tx=0).stop 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f795b7fe640 1 -- 172.21.15.5:0/3353783547 reap_dead start 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 1 -- 172.21.15.5:0/3353783547 shutdown_connections 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 1 -- 172.21.15.5:0/3353783547 >> 172.21.15.5:0/3353783547 conn(0x7f795c0a85d0 msgr2=0x7f795c09db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 1 -- 172.21.15.5:0/3353783547 shutdown_connections 2024-09-13T11:40:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.703+0000 7f79614df640 1 -- 172.21.15.5:0/3353783547 wait complete. 2024-09-13T11:40:15.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 1 -- 172.21.15.5:0/3543121427 --> v2:172.21.15.39:6800/4264535364 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe4000ab260 con 0x7fe3ec040de0 2024-09-13T11:40:15.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe3e77fe640 1 -- 172.21.15.5:0/3543121427 <== osd.4 v2:172.21.15.39:6800/4264535364 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe4000ab260 con 0x7fe3ec040de0 2024-09-13T11:40:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 1 -- 172.21.15.5:0/3543121427 >> v2:172.21.15.39:6800/4264535364 conn(0x7fe3ec040de0 msgr2=0x7fe3ec043220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 1 --2- 172.21.15.5:0/3543121427 >> v2:172.21.15.39:6800/4264535364 conn(0x7fe3ec040de0 0x7fe3ec043220 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 1 -- 172.21.15.5:0/3543121427 >> v2:172.21.15.39:6812/1340838781 conn(0x7fe3ec03b2d0 msgr2=0x7fe3ec03d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 1 --2- 172.21.15.5:0/3543121427 >> v2:172.21.15.39:6812/1340838781 conn(0x7fe3ec03b2d0 0x7fe3ec03d790 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0007fd0 tx=0x7fe3f0007920 comp rx=0 tx=0).stop 2024-09-13T11:40:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 10 monclient: shutdown 2024-09-13T11:40:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 1 -- 172.21.15.5:0/3543121427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe40013cd80 msgr2=0x7fe4001397e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 1 --2- 172.21.15.5:0/3543121427 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe40013cd80 0x7fe4001397e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800c7c0 tx=0x7fe3f800cc80 comp rx=0 tx=0).stop 2024-09-13T11:40:15.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe3ff7fe640 1 -- 172.21.15.5:0/3543121427 reap_dead start 2024-09-13T11:40:15.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 1 -- 172.21.15.5:0/3543121427 shutdown_connections 2024-09-13T11:40:15.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 1 -- 172.21.15.5:0/3543121427 >> 172.21.15.5:0/3543121427 conn(0x7fe4000a6c70 msgr2=0x7fe40013f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 1 -- 172.21.15.5:0/3543121427 shutdown_connections 2024-09-13T11:40:15.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.707+0000 7fe405633640 1 -- 172.21.15.5:0/3543121427 wait complete. 2024-09-13T11:40:15.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.711+0000 7fa52f9ea640 1 -- 172.21.15.5:0/3933300436 --> v2:172.21.15.5:6800/1907498696 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa528120c50 con 0x7fa50c040de0 2024-09-13T11:40:15.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa51dffb640 1 -- 172.21.15.5:0/3933300436 <== osd.2 v2:172.21.15.5:6800/1907498696 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa528120c50 con 0x7fa50c040de0 2024-09-13T11:40:15.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52f9ea640 1 -- 172.21.15.5:0/3933300436 >> v2:172.21.15.5:6800/1907498696 conn(0x7fa50c040de0 msgr2=0x7fa50c043220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52f9ea640 1 --2- 172.21.15.5:0/3933300436 >> v2:172.21.15.5:6800/1907498696 conn(0x7fa50c040de0 0x7fa50c043220 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52f9ea640 1 -- 172.21.15.5:0/3933300436 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa50c03b2d0 msgr2=0x7fa50c03d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52f9ea640 1 --2- 172.21.15.5:0/3933300436 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa50c03b2d0 0x7fa50c03d790 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa5200012f0 tx=0x7fa520005b80 comp rx=0 tx=0).stop 2024-09-13T11:40:15.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52f9ea640 10 monclient: shutdown 2024-09-13T11:40:15.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52f9ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52f9ea640 1 -- 172.21.15.5:0/3933300436 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa528138420 msgr2=0x7fa528139a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52f9ea640 1 --2- 172.21.15.5:0/3933300436 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa528138420 0x7fa528139a40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa518018f50 tx=0x7fa5180190e0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52df87640 1 -- 172.21.15.5:0/3933300436 reap_dead start 2024-09-13T11:40:15.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52f9ea640 1 -- 172.21.15.5:0/3933300436 shutdown_connections 2024-09-13T11:40:15.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52f9ea640 1 -- 172.21.15.5:0/3933300436 >> 172.21.15.5:0/3933300436 conn(0x7fa5280a6c70 msgr2=0x7fa52809fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52f9ea640 1 -- 172.21.15.5:0/3933300436 shutdown_connections 2024-09-13T11:40:15.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.715+0000 7fa52f9ea640 1 -- 172.21.15.5:0/3933300436 wait complete. 2024-09-13T11:40:15.810 INFO:teuthology.orchestra.run.smithi005.stdout:51539607557 2024-09-13T11:40:15.810 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T11:40:15.811 INFO:teuthology.orchestra.run.smithi005.stdout:51539607557 2024-09-13T11:40:15.811 INFO:teuthology.orchestra.run.smithi005.stdout:51539607557 2024-09-13T11:40:15.812 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T11:40:15.812 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T11:40:15.813 INFO:teuthology.orchestra.run.smithi005.stdout:47244640261 2024-09-13T11:40:15.813 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T11:40:15.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.939+0000 7f12f5aa1640 10 monclient: get_monmap_and_config 2024-09-13T11:40:15.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.939+0000 7f12f5aa1640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.939+0000 7f12f5aa1640 10 monclient: monmap: 2024-09-13T11:40:15.942 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.942 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:15.944099+0000 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:15.944099+0000 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.939+0000 7f12f5aa1640 1 Processor -- start 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.939+0000 7f12f5aa1640 1 -- start start 2024-09-13T11:40:15.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.939+0000 7f12f5aa1640 10 monclient: init 2024-09-13T11:40:15.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient: get_monmap_and_config 2024-09-13T11:40:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-13T11:40:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f12f0137930 0x7f12f009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 10 monclient(hunting): picked mon.noname-a con 0x7f12f0137930 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 0x7f12f01380f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 10 monclient(hunting): picked mon.noname-b con 0x7f12f0137d10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12f00a4a70 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12f009bef0 con 0x7f12f00a4a70 2024-09-13T11:40:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f12f00a5fe0 con 0x7f12f0137930 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f12f009c540 con 0x7f12f0137d10 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 10 monclient(hunting): authenticate will time out at 1019.292525s 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 0x7f12f01380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 0x7f12f01380f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59188/0 (socket says 172.21.15.5:59188) 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 1 -- 172.21.15.5:0/3409832181 learned_addr learned my addr 172.21.15.5:0/3409832181 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 10 monclient(hunting): get_auth_request con 0x7f12f0137d10 auth_method 0 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ef7fe640 1 --2- 172.21.15.5:0/3409832181 >> v2:172.21.15.5:3300/0 conn(0x7f12f0137930 0x7f12f009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ef7fe640 10 monclient(hunting): get_auth_request con 0x7f12f0137930 auth_method 0 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 1 -- 172.21.15.5:0/3409832181 >> v2:172.21.15.5:3300/0 conn(0x7f12f0137930 msgr2=0x7f12f009cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 1 --2- 172.21.15.5:0/3409832181 >> v2:172.21.15.5:3300/0 conn(0x7f12f0137930 0x7f12f009cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 1 -- 172.21.15.5:0/3409832181 >> v1:172.21.15.39:6789/0 conn(0x7f12f00a4a70 legacy=0x7f12f00a4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 1 monclient: found mon.noname-b 2024-09-13T11:40:15.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 1 -- 172.21.15.5:0/3409832181 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f009c1d0 con 0x7f12f0137d10 2024-09-13T11:40:15.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient: monmap: 2024-09-13T11:40:15.950 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.950 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.950 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:15.945139+0000 2024-09-13T11:40:15.950 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:15.945139+0000 2024-09-13T11:40:15.950 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.950 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.951 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:15.951 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:15.951 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:15.951 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:15.951 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:15.951 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 1 Processor -- start 2024-09-13T11:40:15.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 1 -- start start 2024-09-13T11:40:15.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient: init 2024-09-13T11:40:15.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-13T11:40:15.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f109809ba40 0x7f109809be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient(hunting): picked mon.noname-b con 0x7f109809ba40 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f109809d1c0 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f109809c360 0x7f109813c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient(hunting): picked mon.noname-a con 0x7f109809c360 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10980a6740 con 0x7f109809d1c0 2024-09-13T11:40:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f109809e730 con 0x7f109809c360 2024-09-13T11:40:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f109809cc50 con 0x7f109809ba40 2024-09-13T11:40:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109e27f640 10 monclient(hunting): authenticate will time out at 1019.294074s 2024-09-13T11:40:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f109809c360 0x7f109813c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f109809c360 0x7f109813c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57320/0 (socket says 172.21.15.5:57320) 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 1 -- 172.21.15.5:0/2282317062 learned_addr learned my addr 172.21.15.5:0/2282317062 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 10 monclient(hunting): get_auth_request con 0x7f109809c360 auth_method 0 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 1 -- 172.21.15.5:0/2282317062 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f109809ba40 msgr2=0x7f109809be20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 1 --2- 172.21.15.5:0/2282317062 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f109809ba40 0x7f109809be20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 1 -- 172.21.15.5:0/2282317062 >> v1:172.21.15.5:6789/0 conn(0x7f109809d1c0 legacy=0x7f109813c100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:15.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 1 monclient: found mon.noname-a 2024-09-13T11:40:15.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f109c81c640 1 -- 172.21.15.5:0/2282317062 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10980a6980 con 0x7f109809c360 2024-09-13T11:40:15.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12eeffd640 1 --2- 172.21.15.5:0/3409832181 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 0x7f12f01380f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f12e40098a0 tx=0x7f12e401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff4e99eb85801a3d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 1 -- 172.21.15.5:0/3409832181 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e40b0070 con 0x7f12f0137d10 2024-09-13T11:40:15.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:15.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 10 monclient: dump: 2024-09-13T11:40:15.955 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.955 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.955 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.955 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 1 -- 172.21.15.5:0/3409832181 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e40ab600 con 0x7f12f0137d10 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 5 monclient: authenticate success, global_id 4250 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12f5aa1640 4 set_mon_vals no callback set 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 1 -- 172.21.15.5:0/3409832181 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e40ab9d0 con 0x7f12f0137d10 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.943+0000 7f12ee7fc640 10 monclient: dump: 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.957 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.958 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient: shutdown 2024-09-13T11:40:15.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 -- 172.21.15.5:0/3409832181 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 msgr2=0x7f12f01380f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 --2- 172.21.15.5:0/3409832181 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 0x7f12f01380f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f12e40098a0 tx=0x7f12e401c400 comp rx=0 tx=0).stop 2024-09-13T11:40:15.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 -- 172.21.15.5:0/3409832181 shutdown_connections 2024-09-13T11:40:15.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 --2- 172.21.15.5:0/3409832181 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 0x7f12f01380f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 --2- 172.21.15.5:0/3409832181 >> v2:172.21.15.5:3300/0 conn(0x7f12f0137930 0x7f12f009cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 -- 172.21.15.5:0/3409832181 >> 172.21.15.5:0/3409832181 conn(0x7f12f00a6c70 msgr2=0x7f12f00a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 -- 172.21.15.5:0/3409832181 shutdown_connections 2024-09-13T11:40:15.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 -- 172.21.15.5:0/3409832181 wait complete. 2024-09-13T11:40:15.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient: monmap: 2024-09-13T11:40:15.959 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.959 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:15.959 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:15.959 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:15.959 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.959 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.959 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:15.959 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:15.959 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:15.959 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 Processor -- start 2024-09-13T11:40:15.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109c81c640 1 --2- 172.21.15.5:0/2282317062 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f109809c360 0x7f109813c810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1090009e50 tx=0x7f109001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0b096997036b0a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 1 -- 172.21.15.5:0/2282317062 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10900a57d0 con 0x7f109809c360 2024-09-13T11:40:15.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:15.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 10 monclient: dump: 2024-09-13T11:40:15.960 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 1 -- 172.21.15.5:0/2282317062 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10900a5970 con 0x7f109809c360 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 1 -- 172.21.15.5:0/2282317062 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10900a5d40 con 0x7f109809c360 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f1096ffd640 10 monclient: dump: 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.962 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.963 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.963 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 5 monclient: authenticate success, global_id 4299 2024-09-13T11:40:15.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:15.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 4 set_mon_vals no callback set 2024-09-13T11:40:15.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 -- start start 2024-09-13T11:40:15.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient: init 2024-09-13T11:40:15.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:40:15.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12f00a4a70 0x7f12f0074910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient(hunting): picked mon.noname-b con 0x7f12f00a4a70 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12f0137930 0x7f12f013cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient(hunting): picked mon.noname-a con 0x7f12f0137930 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 0x7f12f013d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient(hunting): picked mon.noname-c con 0x7f12f0137d10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f12f0142740 con 0x7f12f0137930 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f12f01425c0 con 0x7f12f00a4a70 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f12f01428c0 con 0x7f12f0137d10 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12f5aa1640 10 monclient(hunting): authenticate will time out at 1019.298013s 2024-09-13T11:40:15.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 0x7f12f013d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 0x7f12f013d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59202/0 (socket says 172.21.15.5:59202) 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 1 -- 172.21.15.5:0/1001287018 learned_addr learned my addr 172.21.15.5:0/1001287018 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 10 monclient(hunting): get_auth_request con 0x7f12f0137d10 auth_method 0 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12ef7fe640 1 --2- 172.21.15.5:0/1001287018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12f00a4a70 0x7f12f0074910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12ef7fe640 10 monclient(hunting): get_auth_request con 0x7f12f00a4a70 auth_method 0 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 10 monclient: shutdown 2024-09-13T11:40:15.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:15.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 1 -- 172.21.15.5:0/2282317062 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f109809c360 msgr2=0x7f109813c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 1 --2- 172.21.15.5:0/2282317062 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f109809c360 0x7f109813c810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1090009e50 tx=0x7f109001c970 comp rx=0 tx=0).stop 2024-09-13T11:40:15.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-09-13T11:40:15.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 1 -- 172.21.15.5:0/1001287018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12f00a4a70 msgr2=0x7f12f0074910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 1 --2- 172.21.15.5:0/1001287018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12f00a4a70 0x7f12f0074910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 1 -- 172.21.15.5:0/1001287018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12f0137930 msgr2=0x7f12f013cc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:40:15.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 1 --2- 172.21.15.5:0/1001287018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12f0137930 0x7f12f013cc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 1 monclient: found mon.noname-c 2024-09-13T11:40:15.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 1 -- 172.21.15.5:0/1001287018 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f0071290 con 0x7f12f0137d10 2024-09-13T11:40:15.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 1 -- 172.21.15.5:0/2282317062 shutdown_connections 2024-09-13T11:40:15.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 1 --2- 172.21.15.5:0/2282317062 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f109809c360 0x7f109813c810 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 1 --2- 172.21.15.5:0/2282317062 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f109809ba40 0x7f109809be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 1 -- 172.21.15.5:0/2282317062 >> 172.21.15.5:0/2282317062 conn(0x7f10980a6c70 msgr2=0x7f10980a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:15.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 1 -- 172.21.15.5:0/2282317062 shutdown_connections 2024-09-13T11:40:15.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 1 -- 172.21.15.5:0/2282317062 wait complete. 2024-09-13T11:40:15.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f12effff640 1 --2- 172.21.15.5:0/1001287018 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 0x7f12f013d140 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f12e800c7c0 tx=0x7f12e800cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 10 monclient: monmap: 2024-09-13T11:40:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 1 Processor -- start 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 1 -- start start 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.947+0000 7f109e27f640 10 monclient: init 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f109809ba40 0x7f109813c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient(hunting): picked mon.noname-c con 0x7f109809ba40 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f109809c360 0x7f109813dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient(hunting): picked mon.noname-b con 0x7f109809c360 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f109809d1c0 0x7f109813e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient(hunting): picked mon.noname-a con 0x7f109809d1c0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f109813cd50 con 0x7f109809d1c0 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f109809cfe0 con 0x7f109809c360 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f109813ced0 con 0x7f109809ba40 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient(hunting): authenticate will time out at 1019.300277s 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f109809d1c0 0x7f109813e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f109809d1c0 0x7f109813e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57330/0 (socket says 172.21.15.5:57330) 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 1 -- 172.21.15.5:0/918432390 learned_addr learned my addr 172.21.15.5:0/918432390 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 10 monclient(hunting): get_auth_request con 0x7f109809d1c0 auth_method 0 2024-09-13T11:40:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1097fff640 1 --2- 172.21.15.5:0/918432390 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f109809ba40 0x7f109813c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1097fff640 10 monclient(hunting): get_auth_request con 0x7f109809ba40 auth_method 0 2024-09-13T11:40:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1097fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1097fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1097fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1097fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1097fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1097fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 1 -- 172.21.15.5:0/918432390 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f109809ba40 msgr2=0x7f109813c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 1 --2- 172.21.15.5:0/918432390 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f109809ba40 0x7f109813c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 1 -- 172.21.15.5:0/918432390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f109809c360 msgr2=0x7f109813dd50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 1 --2- 172.21.15.5:0/918432390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f109809c360 0x7f109813dd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 10 monclient: _finish_hunting 0 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 1 monclient: found mon.noname-a 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 1 -- 172.21.15.5:0/918432390 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10981424b0 con 0x7f109809d1c0 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109c81c640 1 --2- 172.21.15.5:0/918432390 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f109809d1c0 0x7f109813e290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10900af040 tx=0x7f1090019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 1 -- 172.21.15.5:0/918432390 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10900bc020 con 0x7f109809d1c0 2024-09-13T11:40:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 10 monclient: dump: 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 1 -- 172.21.15.5:0/918432390 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10900b88c0 con 0x7f109809d1c0 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 5 monclient: authenticate success, global_id 4302 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 1 -- 172.21.15.5:0/918432390 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10900b8cb0 con 0x7f109809d1c0 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient: _renew_subs 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 1 -- 172.21.15.5:0/918432390 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10981417f0 con 0x7f109809d1c0 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 10 monclient: dump: 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.977 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f10967fc640 4 set_mon_vals no callback set 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient: _renew_subs 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 1 -- 172.21.15.5:0/918432390 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f109813cb40 con 0x7f109809d1c0 2024-09-13T11:40:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:15.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:15.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:15.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f109e27f640 1 -- 172.21.15.5:0/918432390 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f109809be20 con 0x7f109809d1c0 2024-09-13T11:40:15.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 1 -- 172.21.15.5:0/918432390 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f10900bf020 con 0x7f109809d1c0 2024-09-13T11:40:15.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 1 --2- 172.21.15.5:0/918432390 >> v2:172.21.15.39:6812/1340838781 conn(0x7f108403b2a0 0x7f108403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1094ff9640 1 -- 172.21.15.5:0/918432390 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f10900e9fa0 con 0x7f109809d1c0 2024-09-13T11:40:15.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1097fff640 1 --2- 172.21.15.5:0/918432390 >> v2:172.21.15.39:6812/1340838781 conn(0x7f108403b2a0 0x7f108403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.951+0000 7f1097fff640 10 monclient: get_auth_request con 0x7f108403b2a0 auth_method 0 2024-09-13T11:40:15.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.955+0000 7f1097fff640 1 --2- 172.21.15.5:0/918432390 >> v2:172.21.15.39:6812/1340838781 conn(0x7f108403b2a0 0x7f108403d760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1088009980 tx=0x7f10880090e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.955+0000 7f1094ff9640 1 -- 172.21.15.5:0/918432390 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f10900c6030 con 0x7f109809d1c0 2024-09-13T11:40:15.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.955+0000 7f1094ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:15.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.955+0000 7f1094ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:15.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.955+0000 7f12d7fff640 1 -- 172.21.15.5:0/1001287018 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8003b80 con 0x7f12f0137d10 2024-09-13T11:40:15.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.955+0000 7f12d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.955+0000 7f12d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.955+0000 7f12d7fff640 10 monclient: dump: 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.955+0000 7f12d7fff640 10 monclient: _finish_auth 0 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.955+0000 7f12d7fff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.955+0000 7f12d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12f5aa1640 5 monclient: authenticate success, global_id 4253 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12f5aa1640 10 monclient: _renew_subs 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12f5aa1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12f5aa1640 1 -- 172.21.15.5:0/1001287018 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f0071570 con 0x7f12f0137d10 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12d7fff640 1 -- 172.21.15.5:0/1001287018 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e8003d20 con 0x7f12f0137d10 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12d7fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12edffb640 4 set_mon_vals no callback set 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12f5aa1640 10 monclient: _renew_subs 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12f5aa1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12f5aa1640 1 -- 172.21.15.5:0/1001287018 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f0070fb0 con 0x7f12f0137d10 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12f5aa1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:15.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12f5aa1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12f5aa1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12f5aa1640 1 -- 172.21.15.5:0/1001287018 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f009cc90 con 0x7f12f0137d10 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12d7fff640 1 -- 172.21.15.5:0/1001287018 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e800aa90 con 0x7f12f0137d10 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12d7fff640 10 monclient: dump: 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:15.984 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:15.985 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:15.985 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:15.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.959+0000 7f12d7fff640 1 -- 172.21.15.5:0/1001287018 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f12e800ac30 con 0x7f12f0137d10 2024-09-13T11:40:15.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.963+0000 7f12d7fff640 1 --2- 172.21.15.5:0/1001287018 >> v2:172.21.15.39:6812/1340838781 conn(0x7f12dc03b2f0 0x7f12dc03d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:15.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.963+0000 7f12d7fff640 1 -- 172.21.15.5:0/1001287018 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f12e804f7a0 con 0x7f12f0137d10 2024-09-13T11:40:15.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.963+0000 7f12ef7fe640 1 --2- 172.21.15.5:0/1001287018 >> v2:172.21.15.39:6812/1340838781 conn(0x7f12dc03b2f0 0x7f12dc03d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:15.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.963+0000 7f12ef7fe640 10 monclient: get_auth_request con 0x7f12dc03b2f0 auth_method 0 2024-09-13T11:40:15.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.963+0000 7f12ef7fe640 1 --2- 172.21.15.5:0/1001287018 >> v2:172.21.15.39:6812/1340838781 conn(0x7f12dc03b2f0 0x7f12dc03d7b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009af0 tx=0x7f12e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:15.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.963+0000 7f12d7fff640 1 -- 172.21.15.5:0/1001287018 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f12e80052b0 con 0x7f12f0137d10 2024-09-13T11:40:15.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.963+0000 7f12d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:15.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.963+0000 7f12d7fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:15.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient: get_monmap_and_config 2024-09-13T11:40:15.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient: build_initial_monmap 2024-09-13T11:40:15.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient: monmap: 2024-09-13T11:40:15.999 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:15.999 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:15.999 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:16.001153+0000 2024-09-13T11:40:15.999 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:16.001153+0000 2024-09-13T11:40:15.999 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:15.999 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.000 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:16.000 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:16.000 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:16.000 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:16.000 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:16.000 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 1 Processor -- start 2024-09-13T11:40:16.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 1 -- start start 2024-09-13T11:40:16.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient: init 2024-09-13T11:40:16.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:16.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-13T11:40:16.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3988137930 0x7f398809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient(hunting): picked mon.noname-a con 0x7f3988137930 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:16.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3988137d10 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:16.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398813cd80 0x7f398813f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient(hunting): picked mon.noname-b con 0x7f398813cd80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:16.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39880a5a80 con 0x7f3988137d10 2024-09-13T11:40:16.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f39880a5fe0 con 0x7f3988137930 2024-09-13T11:40:16.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f398809c540 con 0x7f398813cd80 2024-09-13T11:40:16.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:16.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398ed25640 10 monclient(hunting): authenticate will time out at 1019.349813s 2024-09-13T11:40:16.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398813cd80 0x7f398813f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398813cd80 0x7f398813f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59222/0 (socket says 172.21.15.5:59222) 2024-09-13T11:40:16.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 1 -- 172.21.15.5:0/1405497086 learned_addr learned my addr 172.21.15.5:0/1405497086 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:16.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 10 monclient(hunting): get_auth_request con 0x7f398813cd80 auth_method 0 2024-09-13T11:40:16.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:16.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2024-09-13T11:40:16.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 1 -- 172.21.15.5:0/1405497086 >> v2:172.21.15.5:3300/0 conn(0x7f3988137930 msgr2=0x7f398809cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:16.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 1 --2- 172.21.15.5:0/1405497086 >> v2:172.21.15.5:3300/0 conn(0x7f3988137930 0x7f398809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 1 -- 172.21.15.5:0/1405497086 >> v1:172.21.15.39:6789/0 conn(0x7f3988137d10 legacy=0x7f39881380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 10 monclient: _finish_hunting 0 2024-09-13T11:40:16.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 1 monclient: found mon.noname-b 2024-09-13T11:40:16.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:16.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:40:16.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f398d2c2640 1 -- 172.21.15.5:0/1405497086 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398809bef0 con 0x7f398813cd80 2024-09-13T11:40:16.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient: get_monmap_and_config 2024-09-13T11:40:16.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient: build_initial_monmap 2024-09-13T11:40:16.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient: monmap: 2024-09-13T11:40:16.006 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:16.007 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.007 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:16.002382+0000 2024-09-13T11:40:16.007 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:16.002382+0000 2024-09-13T11:40:16.007 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:16.007 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.007 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:16.007 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:16.007 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:16.007 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:16.007 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:16.007 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 1 Processor -- start 2024-09-13T11:40:16.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 1 -- start start 2024-09-13T11:40:16.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient: init 2024-09-13T11:40:16.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f12cdb49640 10 monclient: get_monmap_and_config 2024-09-13T11:40:16.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f12cdb49640 10 monclient: build_initial_monmap 2024-09-13T11:40:16.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7f12cdb49640 10 monclient: monmap: 2024-09-13T11:40:16.009 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:16.009 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.009 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:16.003767+0000 2024-09-13T11:40:16.009 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:16.003767+0000 2024-09-13T11:40:16.009 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:16.009 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.009 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:16.009 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:16.009 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:16.010 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:16.010 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:16.010 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:16.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-13T11:40:16.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fefa8073420 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:16.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fefa8073f10 0x7fefa813cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient(hunting): picked mon.noname-a con 0x7fefa8073f10 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:16.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefa813d0a0 0x7fefa813f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient(hunting): picked mon.noname-b con 0x7fefa813d0a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:16.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefad451640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fefa8073f10 0x7fefa813cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefad451640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fefa8073f10 0x7fefa813cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57342/0 (socket says 172.21.15.5:57342) 2024-09-13T11:40:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefad451640 1 -- 172.21.15.5:0/1510696311 learned_addr learned my addr 172.21.15.5:0/1510696311 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefadc52640 1 --1- 172.21.15.5:0/1510696311 >> v1:172.21.15.5:6789/0 conn(0x7fefa8073420 0x7fefa8073800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36830/0 (socket says 172.21.15.5:36830) 2024-09-13T11:40:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa80a1ef0 con 0x7fefa8073420 2024-09-13T11:40:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 1 -- 172.21.15.5:0/1510696311 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fefa80a19c0 con 0x7fefa8073f10 2024-09-13T11:40:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefae453640 1 --2- 172.21.15.5:0/1510696311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefa813d0a0 0x7fefa813f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 1 -- 172.21.15.5:0/1510696311 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fefa8074940 con 0x7fefa813d0a0 2024-09-13T11:40:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefafeb6640 10 monclient(hunting): authenticate will time out at 1019.351687s 2024-09-13T11:40:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefad451640 10 monclient(hunting): get_auth_request con 0x7fefa8073f10 auth_method 0 2024-09-13T11:40:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefad451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefad451640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefad451640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefae453640 10 monclient(hunting): get_auth_request con 0x7fefa813d0a0 auth_method 0 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefae453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefae453640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefae453640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefae453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefae453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:15.999+0000 7fefae453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefae453640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefae453640 1 -- 172.21.15.5:0/1510696311 >> v2:172.21.15.5:3300/0 conn(0x7fefa8073f10 msgr2=0x7fefa813cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefae453640 1 --2- 172.21.15.5:0/1510696311 >> v2:172.21.15.5:3300/0 conn(0x7fefa8073f10 0x7fefa813cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefae453640 1 -- 172.21.15.5:0/1510696311 >> v1:172.21.15.5:6789/0 conn(0x7fefa8073420 legacy=0x7fefa8073800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefae453640 10 monclient: _finish_hunting 0 2024-09-13T11:40:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefae453640 1 monclient: found mon.noname-b 2024-09-13T11:40:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefae453640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefae453640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:40:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefae453640 1 -- 172.21.15.5:0/1510696311 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa8074d80 con 0x7fefa813d0a0 2024-09-13T11:40:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 1 -- 172.21.15.5:0/1510696311 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1911847276 0 0) 0x7fefa80a1ef0 con 0x7fefa8073420 2024-09-13T11:40:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:40:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefad451640 1 --2- 172.21.15.5:0/1510696311 >> v2:172.21.15.5:3300/0 conn(0x7fefa8073f10 0x7fefa813cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:40:16.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 1 Processor -- start 2024-09-13T11:40:16.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 1 -- start start 2024-09-13T11:40:16.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 10 monclient: init 2024-09-13T11:40:16.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 10 monclient: get_monmap_and_config 2024-09-13T11:40:16.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 10 monclient: build_initial_monmap 2024-09-13T11:40:16.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 10 monclient: monmap: 2024-09-13T11:40:16.017 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:16.006163+0000 2024-09-13T11:40:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:16.006163+0000 2024-09-13T11:40:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:16.018 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398d2c2640 1 --2- 172.21.15.5:0/1405497086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398813cd80 0x7f398813f170 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3980009870 tx=0x7f398001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4632c786c6c8cefb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:16.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:16.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-13T11:40:16.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c8134990 0x7f12c8134d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 10 monclient(hunting): picked mon.noname-b con 0x7f12c8134990 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:16.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 1 -- 172.21.15.5:0/1405497086 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39800b0070 con 0x7f398813cd80 2024-09-13T11:40:16.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:16.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 10 monclient: dump: 2024-09-13T11:40:16.020 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.020 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.020 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.020 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 5 monclient: authenticate success, global_id 4259 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 1 -- 172.21.15.5:0/1405497086 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39800ab600 con 0x7f398813cd80 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 1 -- 172.21.15.5:0/1405497086 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39800ab9d0 con 0x7f398813cd80 2024-09-13T11:40:16.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:16.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f397f7fe640 10 monclient: dump: 2024-09-13T11:40:16.022 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.022 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.022 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.022 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.022 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.022 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.022 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.022 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.022 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.023 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:16.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 4 set_mon_vals no callback set 2024-09-13T11:40:16.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12c809d2c0 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:16.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f12c809c150 0x7f12c813c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 10 monclient(hunting): picked mon.noname-a con 0x7f12c809c150 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:16.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12c809def0 con 0x7f12c809d2c0 2024-09-13T11:40:16.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f12c80a35d0 con 0x7f12c809c150 2024-09-13T11:40:16.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f12c80a3370 con 0x7f12c8134990 2024-09-13T11:40:16.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:16.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12cdb49640 10 monclient(hunting): authenticate will time out at 1019.353881s 2024-09-13T11:40:16.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f12c809c150 0x7f12c813c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 1 Processor -- start 2024-09-13T11:40:16.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f12c809c150 0x7f12c813c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57348/0 (socket says 172.21.15.5:57348) 2024-09-13T11:40:16.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 1 -- 172.21.15.5:0/676244103 learned_addr learned my addr 172.21.15.5:0/676244103 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:16.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 10 monclient(hunting): get_auth_request con 0x7f12c809c150 auth_method 0 2024-09-13T11:40:16.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 1 -- start start 2024-09-13T11:40:16.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 10 monclient: init 2024-09-13T11:40:16.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:16.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefae453640 1 --2- 172.21.15.5:0/1510696311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefa813d0a0 0x7fefa813f490 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fefa40097c0 tx=0x7fefa401c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=433fc15e10624ac0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:16.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 10 monclient(hunting): handle_auth_done global_id 4311 payload 274 2024-09-13T11:40:16.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 1 -- 172.21.15.5:0/676244103 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c8134990 msgr2=0x7f12c8134d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:16.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 1 --2- 172.21.15.5:0/676244103 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c8134990 0x7f12c8134d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 1 -- 172.21.15.5:0/676244103 >> v1:172.21.15.5:6789/0 conn(0x7f12c809d2c0 legacy=0x7f12c809ba40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:16.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:16.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 1 monclient: found mon.noname-a 2024-09-13T11:40:16.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:16.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:16.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f12c7fff640 1 -- 172.21.15.5:0/676244103 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c809e520 con 0x7f12c809c150 2024-09-13T11:40:16.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 1 -- 172.21.15.5:0/1510696311 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa40b0070 con 0x7fefa813d0a0 2024-09-13T11:40:16.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:16.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 10 monclient: dump: 2024-09-13T11:40:16.028 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.028 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.028 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.028 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.028 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.028 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.028 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.028 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.028 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.029 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 10 monclient: _finish_auth 0 2024-09-13T11:40:16.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 10 monclient: _check_auth_tickets 2024-09-13T11:40:16.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:16.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 1 -- 172.21.15.5:0/1510696311 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefa4004230 con 0x7fefa813d0a0 2024-09-13T11:40:16.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:16.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 1 -- 172.21.15.5:0/1510696311 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa4003620 con 0x7fefa813d0a0 2024-09-13T11:40:16.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:16.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefacc50640 10 monclient: dump: 2024-09-13T11:40:16.029 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.030 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.030 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.030 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.030 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.030 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.030 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.030 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.030 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.030 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefafeb6640 5 monclient: authenticate success, global_id 4262 2024-09-13T11:40:16.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefafeb6640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:16.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefafeb6640 4 set_mon_vals no callback set 2024-09-13T11:40:16.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 10 monclient: shutdown 2024-09-13T11:40:16.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:16.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 1 -- 172.21.15.5:0/1405497086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398813cd80 msgr2=0x7f398813f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 1 --2- 172.21.15.5:0/1405497086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398813cd80 0x7f398813f170 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3980009870 tx=0x7f398001c420 comp rx=0 tx=0).stop 2024-09-13T11:40:16.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 1 -- 172.21.15.5:0/1405497086 shutdown_connections 2024-09-13T11:40:16.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 1 --2- 172.21.15.5:0/1405497086 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398813cd80 0x7f398813f170 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 1 --2- 172.21.15.5:0/1405497086 >> v2:172.21.15.5:3300/0 conn(0x7f3988137930 0x7f398809cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 1 -- 172.21.15.5:0/1405497086 >> 172.21.15.5:0/1405497086 conn(0x7f39880a6c70 msgr2=0x7f39880a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:16.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:16.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-13T11:40:16.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 1 -- 172.21.15.5:0/1405497086 shutdown_connections 2024-09-13T11:40:16.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f398ed25640 1 -- 172.21.15.5:0/1405497086 wait complete. 2024-09-13T11:40:16.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8714137ac0 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:16.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8714137ea0 0x7f871413c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f8713fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f8714137ac0 0x7f871409c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36836/0 (socket says 172.21.15.5:36836) 2024-09-13T11:40:16.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 10 monclient(hunting): picked mon.noname-c con 0x7f8714137ea0 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:16.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f8713fff640 1 -- 172.21.15.5:0/3032073621 learned_addr learned my addr 172.21.15.5:0/3032073621 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:16.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87141382f0 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:16.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 1 -- 172.21.15.5:0/3032073621 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f871406da40 con 0x7f8714137ac0 2024-09-13T11:40:16.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 1 -- 172.21.15.5:0/3032073621 --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8714074c50 con 0x7f87141382f0 2024-09-13T11:40:16.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 1 -- 172.21.15.5:0/3032073621 --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f871409bc00 con 0x7f8714137ea0 2024-09-13T11:40:16.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefafeb6640 10 monclient: shutdown 2024-09-13T11:40:16.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:16.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefafeb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:16.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefafeb6640 1 -- 172.21.15.5:0/1510696311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefa813d0a0 msgr2=0x7fefa813f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefafeb6640 1 --2- 172.21.15.5:0/1510696311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefa813d0a0 0x7fefa813f490 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fefa40097c0 tx=0x7fefa401c330 comp rx=0 tx=0).stop 2024-09-13T11:40:16.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f871a8aa640 10 monclient(hunting): authenticate will time out at 1019.355736s 2024-09-13T11:40:16.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1458062392 0 0) 0x7f871406da40 con 0x7f8714137ac0 2024-09-13T11:40:16.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f87137fe640 10 monclient(hunting): _negotiate 2024-09-13T11:40:16.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f87137fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f87137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f87137fe640 10 monclient(hunting): my global_id is 4314 2024-09-13T11:40:16.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8700003530 con 0x7f8714137ac0 2024-09-13T11:40:16.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefafeb6640 1 -- 172.21.15.5:0/1510696311 shutdown_connections 2024-09-13T11:40:16.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefafeb6640 1 --2- 172.21.15.5:0/1510696311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefa813d0a0 0x7fefa813f490 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefafeb6640 1 --2- 172.21.15.5:0/1510696311 >> v2:172.21.15.5:3300/0 conn(0x7fefa8073f10 0x7fefa813cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7fefafeb6640 1 -- 172.21.15.5:0/1510696311 >> 172.21.15.5:0/1510696311 conn(0x7fefa80a2760 msgr2=0x7fefa80a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:16.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f870bfff640 1 --2- 172.21.15.5:0/3032073621 >> v2:172.21.15.39:3300/0 conn(0x7f8714137ea0 0x7f871413c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f870bfff640 10 monclient(hunting): get_auth_request con 0x7f8714137ea0 auth_method 0 2024-09-13T11:40:16.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f870bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f870bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.003+0000 7f870bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f870bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f870bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f870bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:16.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 1 -- 172.21.15.5:0/1510696311 shutdown_connections 2024-09-13T11:40:16.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 1 -- 172.21.15.5:0/1510696311 wait complete. 2024-09-13T11:40:16.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3082832564 0 0) 0x7f8700003530 con 0x7f8714137ac0 2024-09-13T11:40:16.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 >> v2:172.21.15.39:3300/0 conn(0x7f8714137ea0 msgr2=0x7f871413c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:16.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 --2- 172.21.15.5:0/3032073621 >> v2:172.21.15.39:3300/0 conn(0x7f8714137ea0 0x7f871413c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 >> v1:172.21.15.39:6789/0 conn(0x7f87141382f0 legacy=0x7f871413ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:16.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient: _finish_hunting 0 2024-09-13T11:40:16.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:40:16.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:16.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-13T11:40:16.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871409bee0 con 0x7f8714137ac0 2024-09-13T11:40:16.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:16.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:16.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:16.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3199362888 0 0) 0x7f8704001700 con 0x7f8714137ac0 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient: dump: 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.040 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 >> v1:172.21.15.5:6789/0 conn(0x7f8714137ac0 legacy=0x7f871409c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 --2- 172.21.15.5:0/3032073621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8700003a10 0x7f8700003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient(hunting): picked mon.a con 0x7f8700003a10 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 --2- 172.21.15.5:0/3032073621 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8700004810 0x7f8700004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient(hunting): picked mon.c con 0x7f8700004810 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 --2- 172.21.15.5:0/3032073621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87000052f0 0x7f870000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient(hunting): picked mon.b con 0x7f87000052f0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f871409bc00 con 0x7f8700003a10 2024-09-13T11:40:16.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f870000c1f0 con 0x7f87000052f0 2024-09-13T11:40:16.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f870000c3f0 con 0x7f8700004810 2024-09-13T11:40:16.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:16.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f87137fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:40:16.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient: build_initial_monmap 2024-09-13T11:40:16.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient: monmap: 2024-09-13T11:40:16.042 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:16.043 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:16.043 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:16.043 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:16.043 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:16.043 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.043 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:16.043 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:16.043 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:16.043 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 1 Processor -- start 2024-09-13T11:40:16.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 1 --2- 172.21.15.5:0/3032073621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8700003a10 0x7f8700003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 10 monclient(hunting): get_auth_request con 0x7f8700003a10 auth_method 0 2024-09-13T11:40:16.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:16.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 1 -- start start 2024-09-13T11:40:16.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient: init 2024-09-13T11:40:16.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f870bfff640 1 --2- 172.21.15.5:0/3032073621 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8700004810 0x7f8700004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f870bfff640 10 monclient(hunting): get_auth_request con 0x7f8700004810 auth_method 0 2024-09-13T11:40:16.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f870bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f870bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f870bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:16.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:40:16.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f870bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f870bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f870bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:40:16.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient: build_initial_monmap 2024-09-13T11:40:16.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient: monmap: 2024-09-13T11:40:16.046 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:16.046 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:16.047 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:16.047 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:16.047 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:16.047 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.047 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:16.047 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:16.047 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:16.047 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 1 Processor -- start 2024-09-13T11:40:16.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 1 -- start start 2024-09-13T11:40:16.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient: init 2024-09-13T11:40:16.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 10 monclient(hunting): handle_auth_done global_id 4314 payload 293 2024-09-13T11:40:16.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 1 -- 172.21.15.5:0/3032073621 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8700004810 msgr2=0x7f8700004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:16.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 1 --2- 172.21.15.5:0/3032073621 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8700004810 0x7f8700004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 1 -- 172.21.15.5:0/3032073621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87000052f0 msgr2=0x7f870000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T11:40:16.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 1 --2- 172.21.15.5:0/3032073621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87000052f0 0x7f870000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:16.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 1 monclient: found mon.a 2024-09-13T11:40:16.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:16.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:16.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 1 -- 172.21.15.5:0/3032073621 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8700003530 con 0x7f8700003a10 2024-09-13T11:40:16.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 10 monclient: _finish_auth 0 2024-09-13T11:40:16.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:16.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8713fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:16.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f871a8aa640 5 monclient: authenticate success, global_id 4314 2024-09-13T11:40:16.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f871a8aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:16.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3988137930 0x7f398813c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient(hunting): picked mon.noname-c con 0x7f3988137930 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3988137d10 0x7f3988139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient(hunting): picked mon.noname-b con 0x7f3988137d10 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398813cd80 0x7f3988139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient(hunting): picked mon.noname-a con 0x7f398813cd80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f3988142340 con 0x7f398813cd80 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f39881421c0 con 0x7f3988137d10 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f39881424c0 con 0x7f3988137930 2024-09-13T11:40:16.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:16.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient(hunting): authenticate will time out at 1019.357726s 2024-09-13T11:40:16.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f8718e47640 1 -- 172.21.15.5:0/3032073621 reap_dead start 2024-09-13T11:40:16.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:16.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:40:16.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefa8073420 0x7fefa8120950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient(hunting): picked mon.noname-c con 0x7fefa8073420 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:16.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fefa8073f10 0x7fefa81377c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient(hunting): picked mon.noname-a con 0x7fefa8073f10 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:16.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fefa813d0a0 0x7fefa81cf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient(hunting): picked mon.noname-b con 0x7fefa813d0a0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:16.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fefa80a0e60 con 0x7fefa8073f10 2024-09-13T11:40:16.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fefa80a0ce0 con 0x7fefa813d0a0 2024-09-13T11:40:16.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fefa81421f0 con 0x7fefa8073420 2024-09-13T11:40:16.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:16.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient(hunting): authenticate will time out at 1019.357936s 2024-09-13T11:40:16.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3988137d10 0x7f3988139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398d2c2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398813cd80 0x7f3988139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3988137d10 0x7f3988139970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.5:52896/0 (socket says 172.21.15.5:52896) 2024-09-13T11:40:16.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398d2c2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398813cd80 0x7f3988139eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57370/0 (socket says 172.21.15.5:57370) 2024-09-13T11:40:16.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 1 -- 172.21.15.5:0/476683542 learned_addr learned my addr 172.21.15.5:0/476683542 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:16.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 10 monclient(hunting): get_auth_request con 0x7f3988137d10 auth_method 0 2024-09-13T11:40:16.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefae453640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fefa813d0a0 0x7fefa81cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefae453640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fefa813d0a0 0x7fefa81cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.5:52908/0 (socket says 172.21.15.5:52908) 2024-09-13T11:40:16.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefae453640 1 -- 172.21.15.5:0/1122631148 learned_addr learned my addr 172.21.15.5:0/1122631148 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:16.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefae453640 10 monclient(hunting): get_auth_request con 0x7fefa813d0a0 auth_method 0 2024-09-13T11:40:16.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefae453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefae453640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefae453640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:16.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 1 --2- 172.21.15.5:0/1122631148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fefa8073f10 0x7fefa81377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398d2c2640 10 monclient(hunting): get_auth_request con 0x7f398813cd80 auth_method 0 2024-09-13T11:40:16.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398d2c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398d2c2640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398d2c2640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 10 monclient(hunting): get_auth_request con 0x7fefa8073f10 auth_method 0 2024-09-13T11:40:16.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:16.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398d2c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398d2c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398d2c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:16.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 10 monclient(hunting): handle_auth_done global_id 4303 payload 824 2024-09-13T11:40:16.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 1 -- 172.21.15.5:0/476683542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3988137930 msgr2=0x7f398813c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 1 --2- 172.21.15.5:0/476683542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3988137930 0x7f398813c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 1 -- 172.21.15.5:0/476683542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398813cd80 msgr2=0x7f3988139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 1 --2- 172.21.15.5:0/476683542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398813cd80 0x7f3988139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:16.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 1 monclient: found mon.noname-b 2024-09-13T11:40:16.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:16.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.39:3300/0 2024-09-13T11:40:16.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 1 -- 172.21.15.5:0/476683542 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39880a0f20 con 0x7f3988137d10 2024-09-13T11:40:16.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefae453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefae453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefae453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:16.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-09-13T11:40:16.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 1 -- 172.21.15.5:0/1122631148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefa8073420 msgr2=0x7fefa8120950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 1 --2- 172.21.15.5:0/1122631148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefa8073420 0x7fefa8120950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 1 -- 172.21.15.5:0/1122631148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fefa813d0a0 msgr2=0x7fefa81cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397ffff640 1 --2- 172.21.15.5:0/476683542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3988137d10 0x7f3988139970 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f397800c4b0 tx=0x7f397800c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:16.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 1 --2- 172.21.15.5:0/1122631148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fefa813d0a0 0x7fefa81cf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 10 monclient: _finish_hunting 0 2024-09-13T11:40:16.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 1 monclient: found mon.noname-a 2024-09-13T11:40:16.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:16.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:16.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 1 -- 172.21.15.5:0/1122631148 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa8138460 con 0x7fefa8073f10 2024-09-13T11:40:16.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefad451640 1 --2- 172.21.15.5:0/1122631148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fefa8073f10 0x7fefa81377c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fef9800e810 tx=0x7fef9800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 1 -- 172.21.15.5:0/476683542 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39780052d0 con 0x7f3988137d10 2024-09-13T11:40:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 10 monclient: dump: 2024-09-13T11:40:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.063 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 10 monclient: _finish_auth 0 2024-09-13T11:40:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 10 monclient: _check_auth_tickets 2024-09-13T11:40:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 1 -- 172.21.15.5:0/476683542 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3978005470 con 0x7f3988137d10 2024-09-13T11:40:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 1 -- 172.21.15.5:0/476683542 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3978012850 con 0x7f3988137d10 2024-09-13T11:40:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:40:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397d7fa640 10 monclient: dump: 2024-09-13T11:40:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.064 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.064 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.064 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.064 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.064 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.064 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.064 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.064 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.064 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 1 -- 172.21.15.5:0/1122631148 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9800f040 con 0x7fefa8073f10 2024-09-13T11:40:16.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:16.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 10 monclient: dump: 2024-09-13T11:40:16.065 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 1 -- 172.21.15.5:0/1122631148 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef98004030 con 0x7fefa8073f10 2024-09-13T11:40:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 1 -- 172.21.15.5:0/1122631148 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef98004400 con 0x7fefa8073f10 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9e7fc640 10 monclient: dump: 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.067 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.068 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 5 monclient: authenticate success, global_id 4320 2024-09-13T11:40:16.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient: _renew_subs 2024-09-13T11:40:16.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:16.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 1 -- 172.21.15.5:0/1122631148 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa811ebf0 con 0x7fefa8073f10 2024-09-13T11:40:16.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient: _renew_subs 2024-09-13T11:40:16.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:16.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fefafeb6640 1 -- 172.21.15.5:0/1122631148 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa81218b0 con 0x7fefa8073f10 2024-09-13T11:40:16.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7fef9ffff640 4 set_mon_vals no callback set 2024-09-13T11:40:16.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f8713fff640 1 --2- 172.21.15.5:0/3032073621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8700003a10 0x7f8700003e10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f870400c290 tx=0x7f870401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56134a50c0d641e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:16.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 5 monclient: authenticate success, global_id 4303 2024-09-13T11:40:16.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient: _renew_subs 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 1 -- 172.21.15.5:0/476683542 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39880a1360 con 0x7f3988137d10 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient: _renew_subs 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f398ed25640 1 -- 172.21.15.5:0/476683542 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39880a9210 con 0x7f3988137d10 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.007+0000 7f397effd640 4 set_mon_vals no callback set 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f398ed25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f398ed25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f398ed25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f398ed25640 1 -- 172.21.15.5:0/476683542 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f398809cc90 con 0x7f3988137d10 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f397d7fa640 1 -- 172.21.15.5:0/476683542 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f39780129f0 con 0x7f3988137d10 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f397d7fa640 1 --2- 172.21.15.5:0/476683542 >> v2:172.21.15.39:6812/1340838781 conn(0x7f397403b2a0 0x7f397403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f397d7fa640 1 -- 172.21.15.5:0/476683542 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f397804e880 con 0x7f3988137d10 2024-09-13T11:40:16.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f398cac1640 1 --2- 172.21.15.5:0/476683542 >> v2:172.21.15.39:6812/1340838781 conn(0x7f397403b2a0 0x7f397403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f398cac1640 10 monclient: get_auth_request con 0x7f397403b2a0 auth_method 0 2024-09-13T11:40:16.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7fefafeb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:16.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7fefafeb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:16.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7fefafeb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:16.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7fefafeb6640 1 -- 172.21.15.5:0/1122631148 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef90005090 con 0x7fefa8073f10 2024-09-13T11:40:16.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7fef9e7fc640 1 -- 172.21.15.5:0/1122631148 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fef9800c6d0 con 0x7fefa8073f10 2024-09-13T11:40:16.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704007d60 con 0x7f8700003a10 2024-09-13T11:40:16.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f87137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f87137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:16.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f87137fe640 10 monclient: dump: 2024-09-13T11:40:16.074 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.074 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.074 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.075 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.075 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.075 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.075 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.075 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.075 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.075 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87040a9020 con 0x7f8700003a10 2024-09-13T11:40:16.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f871a8aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:16.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f87137fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:16.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f871a8aa640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f871a8aa640 4 set_mon_vals no callback set 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f87137fe640 1 -- 172.21.15.5:0/3032073621 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704005b20 con 0x7f8700003a10 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f87137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f87137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f87137fe640 10 monclient: dump: 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.076 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.077 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.077 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f398cac1640 1 --2- 172.21.15.5:0/476683542 >> v2:172.21.15.39:6812/1340838781 conn(0x7f397403b2a0 0x7f397403d760 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f398400ab90 tx=0x7f3984009250 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:16.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7fef9e7fc640 1 --2- 172.21.15.5:0/1122631148 >> v2:172.21.15.39:6812/1340838781 conn(0x7fef8c03b2a0 0x7fef8c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7fef9e7fc640 1 -- 172.21.15.5:0/1122631148 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fef98052120 con 0x7fefa8073f10 2024-09-13T11:40:16.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f12c7fff640 1 --2- 172.21.15.5:0/676244103 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c809c150 0x7f12c813c8a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f12b800b1a0 tx=0x7f12b801cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ac7ba1df2203cc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:16.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7fefadc52640 1 --2- 172.21.15.5:0/1122631148 >> v2:172.21.15.39:6812/1340838781 conn(0x7fef8c03b2a0 0x7fef8c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7fefadc52640 10 monclient: get_auth_request con 0x7fef8c03b2a0 auth_method 0 2024-09-13T11:40:16.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f871a8aa640 10 monclient: shutdown 2024-09-13T11:40:16.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f871a8aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:16.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f871a8aa640 1 -- 172.21.15.5:0/3032073621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8700003a10 msgr2=0x7f8700003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f871a8aa640 1 --2- 172.21.15.5:0/3032073621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8700003a10 0x7f8700003e10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f870400c290 tx=0x7f870401cc60 comp rx=0 tx=0).stop 2024-09-13T11:40:16.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f871a8aa640 1 -- 172.21.15.5:0/3032073621 shutdown_connections 2024-09-13T11:40:16.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f871a8aa640 1 --2- 172.21.15.5:0/3032073621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8700003a10 0x7f8700003e10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f871a8aa640 1 -- 172.21.15.5:0/3032073621 >> 172.21.15.5:0/3032073621 conn(0x7f87140a7aa0 msgr2=0x7f87140a3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:16.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f871a8aa640 1 -- 172.21.15.5:0/3032073621 shutdown_connections 2024-09-13T11:40:16.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7fefadc52640 1 --2- 172.21.15.5:0/1122631148 >> v2:172.21.15.39:6812/1340838781 conn(0x7fef8c03b2a0 0x7fef8c03d760 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fefa0000f90 tx=0x7fefa0007c60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:16.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.011+0000 7f871a8aa640 1 -- 172.21.15.5:0/3032073621 wait complete. 2024-09-13T11:40:16.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient: build_initial_monmap 2024-09-13T11:40:16.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient: monmap: 2024-09-13T11:40:16.083 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:16.083 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:16.083 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:16.083 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:16.083 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:16.083 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.083 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:16.083 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:16.083 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 1 Processor -- start 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 1 -- start start 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient: init 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8700003a10 0x7f871413c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient(hunting): picked mon.noname-a con 0x7f8700003a10 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f871413c7c0 0x7f871413cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient(hunting): picked mon.noname-c con 0x7f871413c7c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87141dc7f0 0x7f87141dcc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient(hunting): picked mon.noname-b con 0x7f87141dc7f0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:16.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f87140a3490 con 0x7f8700003a10 2024-09-13T11:40:16.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f87140a3310 con 0x7f87141dc7f0 2024-09-13T11:40:16.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f87140a3610 con 0x7f871413c7c0 2024-09-13T11:40:16.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:16.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient(hunting): authenticate will time out at 1019.365886s 2024-09-13T11:40:16.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f871413c7c0 0x7f871413cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f871413c7c0 0x7f871413cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59288/0 (socket says 172.21.15.5:59288) 2024-09-13T11:40:16.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 1 -- 172.21.15.5:0/1342065882 learned_addr learned my addr 172.21.15.5:0/1342065882 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:16.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 10 monclient(hunting): get_auth_request con 0x7f871413c7c0 auth_method 0 2024-09-13T11:40:16.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8718e47640 1 --2- 172.21.15.5:0/1342065882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87141dc7f0 0x7f87141dcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8718e47640 10 monclient(hunting): get_auth_request con 0x7f87141dc7f0 auth_method 0 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8718e47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8718e47640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8718e47640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8713fff640 1 --2- 172.21.15.5:0/1342065882 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8700003a10 0x7f871413c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8713fff640 10 monclient(hunting): get_auth_request con 0x7f8700003a10 auth_method 0 2024-09-13T11:40:16.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8713fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8718e47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8718e47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8718e47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 1 -- 172.21.15.5:0/1342065882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87141dc7f0 msgr2=0x7f87141dcc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 1 --2- 172.21.15.5:0/1342065882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87141dc7f0 0x7f87141dcc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 1 -- 172.21.15.5:0/1342065882 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8700003a10 msgr2=0x7f871413c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 1 --2- 172.21.15.5:0/1342065882 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8700003a10 0x7f871413c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 1 monclient: found mon.noname-c 2024-09-13T11:40:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 1 -- 172.21.15.5:0/1342065882 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87141374b0 con 0x7f871413c7c0 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f870bfff640 1 --2- 172.21.15.5:0/1342065882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f871413c7c0 0x7f871413cc20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0119d0 tx=0x7f86fc011e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 1 -- 172.21.15.5:0/1342065882 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc00fb50 con 0x7f871413c7c0 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 10 monclient: dump: 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 10 monclient: _finish_auth 0 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 10 monclient: _check_auth_tickets 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 1 -- 172.21.15.5:0/1342065882 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86fc00fd10 con 0x7f871413c7c0 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 1 -- 172.21.15.5:0/1342065882 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc014680 con 0x7f871413c7c0 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f87117fa640 10 monclient: dump: 2024-09-13T11:40:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f8712ffd640 4 set_mon_vals no callback set 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 5 monclient: authenticate success, global_id 4265 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient: _renew_subs 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.015+0000 7f871a8aa640 1 -- 172.21.15.5:0/1342065882 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8714140360 con 0x7f871413c7c0 2024-09-13T11:40:16.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f87117fa640 1 -- 172.21.15.5:0/1342065882 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f86fc012040 con 0x7f871413c7c0 2024-09-13T11:40:16.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f87117fa640 1 --2- 172.21.15.5:0/1342065882 >> v2:172.21.15.39:6812/1340838781 conn(0x7f86f403b2f0 0x7f86f403d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f871a8aa640 10 monclient: _renew_subs 2024-09-13T11:40:16.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f871a8aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:16.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f871a8aa640 1 -- 172.21.15.5:0/1342065882 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f871413dc70 con 0x7f871413c7c0 2024-09-13T11:40:16.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f87117fa640 1 -- 172.21.15.5:0/1342065882 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f86fc0133a0 con 0x7f871413c7c0 2024-09-13T11:40:16.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f8713fff640 1 --2- 172.21.15.5:0/1342065882 >> v2:172.21.15.39:6812/1340838781 conn(0x7f86f403b2f0 0x7f86f403d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f8713fff640 10 monclient: get_auth_request con 0x7f86f403b2f0 auth_method 0 2024-09-13T11:40:16.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f870a7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:16.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f870a7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:16.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f870a7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:16.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f870a7fc640 1 -- 172.21.15.5:0/1342065882 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86ec005090 con 0x7f871413c7c0 2024-09-13T11:40:16.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7fef9e7fc640 1 -- 172.21.15.5:0/1122631148 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fef98017020 con 0x7fefa8073f10 2024-09-13T11:40:16.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7fef9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:16.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7fef9e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:16.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.019+0000 7f8713fff640 1 --2- 172.21.15.5:0/1342065882 >> v2:172.21.15.39:6812/1340838781 conn(0x7f86f403b2f0 0x7f86f403d7b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f87040aa000 tx=0x7f8714074e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 1 -- 172.21.15.5:0/676244103 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b80a58c0 con 0x7f12c809c150 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 10 monclient: dump: 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.093 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.094 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:16.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:16.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:16.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12cdb49640 5 monclient: authenticate success, global_id 4311 2024-09-13T11:40:16.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12cdb49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:16.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f397d7fa640 1 -- 172.21.15.5:0/476683542 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f39880a1360 con 0x7f3988137d10 2024-09-13T11:40:16.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f397d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:16.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f397d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:16.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 1 -- 172.21.15.5:0/676244103 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12b80a5a60 con 0x7f12c809c150 2024-09-13T11:40:16.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:16.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 1 -- 172.21.15.5:0/676244103 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b80a5e30 con 0x7f12c809c150 2024-09-13T11:40:16.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12c67fc640 10 monclient: dump: 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12cdb49640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:16.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12cdb49640 4 set_mon_vals no callback set 2024-09-13T11:40:16.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12cdb49640 10 monclient: shutdown 2024-09-13T11:40:16.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12cdb49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:16.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12cdb49640 1 -- 172.21.15.5:0/676244103 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c809c150 msgr2=0x7f12c813c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.023+0000 7f12cdb49640 1 --2- 172.21.15.5:0/676244103 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c809c150 0x7f12c813c8a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f12b800b1a0 tx=0x7f12b801cb50 comp rx=0 tx=0).stop 2024-09-13T11:40:16.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 -- 172.21.15.5:0/676244103 shutdown_connections 2024-09-13T11:40:16.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 --2- 172.21.15.5:0/676244103 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c809c150 0x7f12c813c8a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 --2- 172.21.15.5:0/676244103 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c8134990 0x7f12c8134d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 -- 172.21.15.5:0/676244103 >> 172.21.15.5:0/676244103 conn(0x7f12c80a3980 msgr2=0x7f12c80a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:16.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 -- 172.21.15.5:0/676244103 shutdown_connections 2024-09-13T11:40:16.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 -- 172.21.15.5:0/676244103 wait complete. 2024-09-13T11:40:16.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient: build_initial_monmap 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient: monmap: 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 Processor -- start 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 -- start start 2024-09-13T11:40:16.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient: init 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12c809c150 0x7f12c81dc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient(hunting): picked mon.noname-b con 0x7f12c809c150 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c809d2c0 0x7f12c8133ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient(hunting): picked mon.noname-a con 0x7f12c809d2c0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c8134990 0x7f12c8134010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient(hunting): picked mon.noname-c con 0x7f12c8134990 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f12c809ed60 con 0x7f12c809d2c0 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f12c809ebe0 con 0x7f12c809c150 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f12c806db20 con 0x7f12c8134990 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12cdb49640 10 monclient(hunting): authenticate will time out at 1019.378662s 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c809d2c0 0x7f12c8133ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12c809c150 0x7f12c81dc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c809d2c0 0x7f12c8133ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57404/0 (socket says 172.21.15.5:57404) 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c6ffd640 1 -- 172.21.15.5:0/3857254954 learned_addr learned my addr 172.21.15.5:0/3857254954 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c6ffd640 10 monclient(hunting): get_auth_request con 0x7f12c809d2c0 auth_method 0 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 10 monclient(hunting): get_auth_request con 0x7f12c809c150 auth_method 0 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 10 monclient(hunting): handle_auth_done global_id 4312 payload 824 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 1 -- 172.21.15.5:0/3857254954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c8134990 msgr2=0x7f12c8134010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 1 --2- 172.21.15.5:0/3857254954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c8134990 0x7f12c8134010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 1 -- 172.21.15.5:0/3857254954 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c809d2c0 msgr2=0x7f12c8133ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 1 --2- 172.21.15.5:0/3857254954 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c809d2c0 0x7f12c8133ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 10 monclient: _finish_hunting 0 2024-09-13T11:40:16.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 1 monclient: found mon.noname-b 2024-09-13T11:40:16.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:16.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.39:3300/0 2024-09-13T11:40:16.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 1 -- 172.21.15.5:0/3857254954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c812ffe0 con 0x7f12c809c150 2024-09-13T11:40:16.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f87117fa640 1 -- 172.21.15.5:0/1342065882 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f86fc040070 con 0x7f871413c7c0 2024-09-13T11:40:16.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f87117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:16.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f87117fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.027+0000 7f12c77fe640 1 --2- 172.21.15.5:0/3857254954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12c809c150 0x7f12c81dc750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00b420 tx=0x7f12bc00b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 1 -- 172.21.15.5:0/3857254954 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12bc011830 con 0x7f12c809c150 2024-09-13T11:40:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 10 monclient: dump: 2024-09-13T11:40:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 10 monclient: _finish_auth 0 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 1 -- 172.21.15.5:0/3857254954 <== mon.1 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12bc016070 con 0x7f12c809c150 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12c5ffb640 4 set_mon_vals no callback set 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 1 -- 172.21.15.5:0/3857254954 <== mon.1 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12bc011c30 con 0x7f12c809c150 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12cdb49640 5 monclient: authenticate success, global_id 4312 2024-09-13T11:40:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12cdb49640 10 monclient: _renew_subs 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12cdb49640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12cdb49640 1 -- 172.21.15.5:0/3857254954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12c81dcc90 con 0x7f12c809c150 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 10 monclient: dump: 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12cdb49640 10 monclient: _renew_subs 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12cdb49640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12cdb49640 1 -- 172.21.15.5:0/3857254954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12c80a6840 con 0x7f12c809c150 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12cdb49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12cdb49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12cdb49640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12cdb49640 1 -- 172.21.15.5:0/3857254954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12c809e0e0 con 0x7f12c809c150 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 1 -- 172.21.15.5:0/3857254954 <== mon.1 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f12bc01b460 con 0x7f12c809c150 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 1 --2- 172.21.15.5:0/3857254954 >> v2:172.21.15.39:6812/1340838781 conn(0x7f12b403b2a0 0x7f12b403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12affff640 1 -- 172.21.15.5:0/3857254954 <== mon.1 v2:172.21.15.39:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f12bc04e8f0 con 0x7f12c809c150 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12c6ffd640 1 --2- 172.21.15.5:0/3857254954 >> v2:172.21.15.39:6812/1340838781 conn(0x7f12b403b2a0 0x7f12b403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.031+0000 7f12c6ffd640 10 monclient: get_auth_request con 0x7f12b403b2a0 auth_method 0 2024-09-13T11:40:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.035+0000 7f12c6ffd640 1 --2- 172.21.15.5:0/3857254954 >> v2:172.21.15.39:6812/1340838781 conn(0x7f12b403b2a0 0x7f12b403d760 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f12c8135100 tx=0x7f12c0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:16.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.047+0000 7f12affff640 1 -- 172.21.15.5:0/3857254954 <== mon.1 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f12bc011dd0 con 0x7f12c809c150 2024-09-13T11:40:16.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.047+0000 7f12affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:16.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.047+0000 7f12affff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:16.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.155+0000 7f109e27f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T11:40:16.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.155+0000 7f109e27f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T11:40:16.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.155+0000 7f109e27f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:16.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.155+0000 7f109e27f640 1 -- 172.21.15.5:0/918432390 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1098143fc0 con 0x7f109809d1c0 2024-09-13T11:40:16.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.155+0000 7f1094ff9640 1 -- 172.21.15.5:0/918432390 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f10981424b0 con 0x7f109809d1c0 2024-09-13T11:40:16.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.155+0000 7f1094ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T11:40:16.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.155+0000 7f1094ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:16.159 INFO:teuthology.orchestra.run.smithi005.stdout:51539607555 2024-09-13T11:40:16.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 1 -- 172.21.15.5:0/918432390 >> v2:172.21.15.39:6812/1340838781 conn(0x7f108403b2a0 msgr2=0x7f108403d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 1 --2- 172.21.15.5:0/918432390 >> v2:172.21.15.39:6812/1340838781 conn(0x7f108403b2a0 0x7f108403d760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1088009980 tx=0x7f10880090e0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 10 monclient: shutdown 2024-09-13T11:40:16.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:16.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 1 -- 172.21.15.5:0/918432390 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f109809d1c0 msgr2=0x7f109813e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 1 --2- 172.21.15.5:0/918432390 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f109809d1c0 0x7f109813e290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10900af040 tx=0x7f1090019710 comp rx=0 tx=0).stop 2024-09-13T11:40:16.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 1 -- 172.21.15.5:0/918432390 shutdown_connections 2024-09-13T11:40:16.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 1 --2- 172.21.15.5:0/918432390 >> v2:172.21.15.39:6812/1340838781 conn(0x7f108403b2a0 0x7f108403d760 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 1 --2- 172.21.15.5:0/918432390 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f109809d1c0 0x7f109813e290 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 1 --2- 172.21.15.5:0/918432390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f109809c360 0x7f109813dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 1 --2- 172.21.15.5:0/918432390 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f109809ba40 0x7f109813c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 1 -- 172.21.15.5:0/918432390 >> 172.21.15.5:0/918432390 conn(0x7f10980a6c70 msgr2=0x7f10980a2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:16.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 1 -- 172.21.15.5:0/918432390 shutdown_connections 2024-09-13T11:40:16.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.159+0000 7f109e27f640 1 -- 172.21.15.5:0/918432390 wait complete. 2024-09-13T11:40:16.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.163+0000 7f12f5aa1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T11:40:16.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.163+0000 7f12f5aa1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T11:40:16.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.163+0000 7f12f5aa1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:16.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.163+0000 7f12f5aa1640 1 -- 172.21.15.5:0/1001287018 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f12f013eee0 con 0x7f12f0137d10 2024-09-13T11:40:16.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.163+0000 7f12d7fff640 1 -- 172.21.15.5:0/1001287018 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f12e804d670 con 0x7f12f0137d10 2024-09-13T11:40:16.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.163+0000 7f12d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T11:40:16.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.163+0000 7f12d7fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:16.165 INFO:teuthology.orchestra.run.smithi005.stdout:51539607555 2024-09-13T11:40:16.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 1 -- 172.21.15.5:0/1001287018 >> v2:172.21.15.39:6812/1340838781 conn(0x7f12dc03b2f0 msgr2=0x7f12dc03d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 1 --2- 172.21.15.5:0/1001287018 >> v2:172.21.15.39:6812/1340838781 conn(0x7f12dc03b2f0 0x7f12dc03d7b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009af0 tx=0x7f12e0008040 comp rx=0 tx=0).stop 2024-09-13T11:40:16.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 10 monclient: shutdown 2024-09-13T11:40:16.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:16.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 1 -- 172.21.15.5:0/1001287018 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 msgr2=0x7f12f013d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 1 --2- 172.21.15.5:0/1001287018 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 0x7f12f013d140 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f12e800c7c0 tx=0x7f12e800cc80 comp rx=0 tx=0).stop 2024-09-13T11:40:16.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 1 -- 172.21.15.5:0/1001287018 shutdown_connections 2024-09-13T11:40:16.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 1 --2- 172.21.15.5:0/1001287018 >> v2:172.21.15.39:6812/1340838781 conn(0x7f12dc03b2f0 0x7f12dc03d7b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 1 --2- 172.21.15.5:0/1001287018 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12f0137d10 0x7f12f013d140 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 1 --2- 172.21.15.5:0/1001287018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12f0137930 0x7f12f013cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 1 --2- 172.21.15.5:0/1001287018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12f00a4a70 0x7f12f0074910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 1 -- 172.21.15.5:0/1001287018 >> 172.21.15.5:0/1001287018 conn(0x7f12f00a6c70 msgr2=0x7f12f00a32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:16.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 1 -- 172.21.15.5:0/1001287018 shutdown_connections 2024-09-13T11:40:16.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.167+0000 7f12f5aa1640 1 -- 172.21.15.5:0/1001287018 wait complete. 2024-09-13T11:40:16.179 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.5 2024-09-13T11:40:16.190 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.1 2024-09-13T11:40:16.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.287+0000 7f398ed25640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T11:40:16.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.287+0000 7f398ed25640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T11:40:16.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.287+0000 7f398ed25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:40:16.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.287+0000 7f398ed25640 1 -- 172.21.15.5:0/476683542 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f398813cac0 con 0x7f3988137d10 2024-09-13T11:40:16.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.287+0000 7f397d7fa640 1 -- 172.21.15.5:0/476683542 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f39880a0f20 con 0x7f3988137d10 2024-09-13T11:40:16.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.287+0000 7f397d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T11:40:16.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.287+0000 7f397d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:16.290 INFO:teuthology.orchestra.run.smithi005.stdout:47244640260 2024-09-13T11:40:16.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 1 -- 172.21.15.5:0/476683542 >> v2:172.21.15.39:6812/1340838781 conn(0x7f397403b2a0 msgr2=0x7f397403d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 1 --2- 172.21.15.5:0/476683542 >> v2:172.21.15.39:6812/1340838781 conn(0x7f397403b2a0 0x7f397403d760 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f398400ab90 tx=0x7f3984009250 comp rx=0 tx=0).stop 2024-09-13T11:40:16.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 10 monclient: shutdown 2024-09-13T11:40:16.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 1 -- 172.21.15.5:0/476683542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3988137d10 msgr2=0x7f3988139970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 1 --2- 172.21.15.5:0/476683542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3988137d10 0x7f3988139970 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f397800c4b0 tx=0x7f397800c970 comp rx=0 tx=0).stop 2024-09-13T11:40:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 1 -- 172.21.15.5:0/476683542 shutdown_connections 2024-09-13T11:40:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 1 --2- 172.21.15.5:0/476683542 >> v2:172.21.15.39:6812/1340838781 conn(0x7f397403b2a0 0x7f397403d760 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 1 --2- 172.21.15.5:0/476683542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398813cd80 0x7f3988139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 1 --2- 172.21.15.5:0/476683542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3988137d10 0x7f3988139970 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 1 --2- 172.21.15.5:0/476683542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3988137930 0x7f398813c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 1 -- 172.21.15.5:0/476683542 >> 172.21.15.5:0/476683542 conn(0x7f39880a6c70 msgr2=0x7f398813ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 1 -- 172.21.15.5:0/476683542 shutdown_connections 2024-09-13T11:40:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.291+0000 7f398ed25640 1 -- 172.21.15.5:0/476683542 wait complete. 2024-09-13T11:40:16.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.299+0000 7fefafeb6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T11:40:16.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.299+0000 7fefafeb6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T11:40:16.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.299+0000 7fefafeb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:16.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.299+0000 7fefafeb6640 1 -- 172.21.15.5:0/1122631148 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fef90005380 con 0x7fefa8073f10 2024-09-13T11:40:16.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.299+0000 7fef9e7fc640 1 -- 172.21.15.5:0/1122631148 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fef9804d910 con 0x7fefa8073f10 2024-09-13T11:40:16.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.299+0000 7fef9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T11:40:16.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.299+0000 7fef9e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:16.302 INFO:teuthology.orchestra.run.smithi005.stdout:51539607555 2024-09-13T11:40:16.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 1 -- 172.21.15.5:0/1122631148 >> v2:172.21.15.39:6812/1340838781 conn(0x7fef8c03b2a0 msgr2=0x7fef8c03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 1 --2- 172.21.15.5:0/1122631148 >> v2:172.21.15.39:6812/1340838781 conn(0x7fef8c03b2a0 0x7fef8c03d760 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fefa0000f90 tx=0x7fefa0007c60 comp rx=0 tx=0).stop 2024-09-13T11:40:16.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 10 monclient: shutdown 2024-09-13T11:40:16.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:16.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 1 -- 172.21.15.5:0/1122631148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fefa8073f10 msgr2=0x7fefa81377c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 1 --2- 172.21.15.5:0/1122631148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fefa8073f10 0x7fefa81377c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fef9800e810 tx=0x7fef9800ecd0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 1 -- 172.21.15.5:0/1122631148 shutdown_connections 2024-09-13T11:40:16.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 1 --2- 172.21.15.5:0/1122631148 >> v2:172.21.15.39:6812/1340838781 conn(0x7fef8c03b2a0 0x7fef8c03d760 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 1 --2- 172.21.15.5:0/1122631148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fefa813d0a0 0x7fefa81cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 1 --2- 172.21.15.5:0/1122631148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fefa8073f10 0x7fefa81377c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 1 --2- 172.21.15.5:0/1122631148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fefa8073420 0x7fefa8120950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 1 -- 172.21.15.5:0/1122631148 >> 172.21.15.5:0/1122631148 conn(0x7fefa80a2760 msgr2=0x7fefa80a49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:16.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 1 -- 172.21.15.5:0/1122631148 shutdown_connections 2024-09-13T11:40:16.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.303+0000 7fefafeb6640 1 -- 172.21.15.5:0/1122631148 wait complete. 2024-09-13T11:40:16.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T11:40:16.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T11:40:16.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.39:3300/0 2024-09-13T11:40:16.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 -- 172.21.15.5:0/3857254954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f12c806df80 con 0x7f12c809c150 2024-09-13T11:40:16.307 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.2 2024-09-13T11:40:16.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12affff640 1 -- 172.21.15.5:0/3857254954 <== mon.1 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f12bc01a3c0 con 0x7f12c809c150 2024-09-13T11:40:16.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T11:40:16.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12affff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:16.308 INFO:teuthology.orchestra.run.smithi005.stdout:47244640260 2024-09-13T11:40:16.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f870a7fc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T11:40:16.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f870a7fc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T11:40:16.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f870a7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:16.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f870a7fc640 1 -- 172.21.15.5:0/1342065882 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f86ec005650 con 0x7f871413c7c0 2024-09-13T11:40:16.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f87117fa640 1 -- 172.21.15.5:0/1342065882 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f86fc019030 con 0x7f871413c7c0 2024-09-13T11:40:16.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f87117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T11:40:16.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f87117fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:16.309 INFO:teuthology.orchestra.run.smithi005.stdout:51539607555 2024-09-13T11:40:16.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 -- 172.21.15.5:0/3857254954 >> v2:172.21.15.39:6812/1340838781 conn(0x7f12b403b2a0 msgr2=0x7f12b403d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 --2- 172.21.15.5:0/3857254954 >> v2:172.21.15.39:6812/1340838781 conn(0x7f12b403b2a0 0x7f12b403d760 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f12c8135100 tx=0x7f12c0009190 comp rx=0 tx=0).stop 2024-09-13T11:40:16.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 10 monclient: shutdown 2024-09-13T11:40:16.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:16.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 -- 172.21.15.5:0/3857254954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12c809c150 msgr2=0x7f12c81dc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 --2- 172.21.15.5:0/3857254954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12c809c150 0x7f12c81dc750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00b420 tx=0x7f12bc00b8e0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 -- 172.21.15.5:0/3857254954 shutdown_connections 2024-09-13T11:40:16.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 --2- 172.21.15.5:0/3857254954 >> v2:172.21.15.39:6812/1340838781 conn(0x7f12b403b2a0 0x7f12b403d760 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 --2- 172.21.15.5:0/3857254954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12c8134990 0x7f12c8134010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 --2- 172.21.15.5:0/3857254954 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12c809d2c0 0x7f12c8133ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 --2- 172.21.15.5:0/3857254954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12c809c150 0x7f12c81dc750 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 -- 172.21.15.5:0/3857254954 >> 172.21.15.5:0/3857254954 conn(0x7f12c80a3980 msgr2=0x7f12c813edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:16.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 -- 172.21.15.5:0/3857254954 shutdown_connections 2024-09-13T11:40:16.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.307+0000 7f12cdb49640 1 -- 172.21.15.5:0/3857254954 wait complete. 2024-09-13T11:40:16.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 1 -- 172.21.15.5:0/1342065882 >> v2:172.21.15.39:6812/1340838781 conn(0x7f86f403b2f0 msgr2=0x7f86f403d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 1 --2- 172.21.15.5:0/1342065882 >> v2:172.21.15.39:6812/1340838781 conn(0x7f86f403b2f0 0x7f86f403d7b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f87040aa000 tx=0x7f8714074e20 comp rx=0 tx=0).stop 2024-09-13T11:40:16.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 10 monclient: shutdown 2024-09-13T11:40:16.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:16.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 1 -- 172.21.15.5:0/1342065882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f871413c7c0 msgr2=0x7f871413cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:16.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 1 --2- 172.21.15.5:0/1342065882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f871413c7c0 0x7f871413cc20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0119d0 tx=0x7f86fc011e90 comp rx=0 tx=0).stop 2024-09-13T11:40:16.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 1 -- 172.21.15.5:0/1342065882 shutdown_connections 2024-09-13T11:40:16.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 1 --2- 172.21.15.5:0/1342065882 >> v2:172.21.15.39:6812/1340838781 conn(0x7f86f403b2f0 0x7f86f403d7b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 1 --2- 172.21.15.5:0/1342065882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87141dc7f0 0x7f87141dcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 1 --2- 172.21.15.5:0/1342065882 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f871413c7c0 0x7f871413cc20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 1 --2- 172.21.15.5:0/1342065882 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8700003a10 0x7f871413c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:16.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 1 -- 172.21.15.5:0/1342065882 >> 172.21.15.5:0/1342065882 conn(0x7f87140a7aa0 msgr2=0x7f87140a0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:16.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 1 -- 172.21.15.5:0/1342065882 shutdown_connections 2024-09-13T11:40:16.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:16.311+0000 7f870a7fc640 1 -- 172.21.15.5:0/1342065882 wait complete. 2024-09-13T11:40:16.324 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.3 2024-09-13T11:40:16.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.0 2024-09-13T11:40:16.330 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.4 2024-09-13T11:40:17.180 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T11:40:17.190 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T11:40:17.308 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T11:40:17.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.307+0000 7f0d39abc640 10 monclient: get_monmap_and_config 2024-09-13T11:40:17.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.307+0000 7f0d39abc640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.307+0000 7f0d39abc640 10 monclient: monmap: 2024-09-13T11:40:17.311 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.311 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.311 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:17.312524+0000 2024-09-13T11:40:17.311 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:17.312524+0000 2024-09-13T11:40:17.311 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.311 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.312 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:17.312 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:17.312 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:17.312 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:17.312 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:17.312 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 1 Processor -- start 2024-09-13T11:40:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 1 -- start start 2024-09-13T11:40:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 10 monclient: init 2024-09-13T11:40:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-13T11:40:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a5b00 0x7f0d34073e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 10 monclient(hunting): picked mon.noname-b con 0x7f0d340a5b00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d340a3e70 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0d3413a720 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d340a3c30 con 0x7f0d3413a720 2024-09-13T11:40:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d340a56b0 con 0x7f0d340a3e70 2024-09-13T11:40:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f0d34073520 con 0x7f0d340a5b00 2024-09-13T11:40:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d39abc640 10 monclient(hunting): authenticate will time out at 1020.661540s 2024-09-13T11:40:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a5b00 0x7f0d34073e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a5b00 0x7f0d34073e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59294/0 (socket says 172.21.15.5:59294) 2024-09-13T11:40:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 1 -- 172.21.15.5:0/2151988100 learned_addr learned my addr 172.21.15.5:0/2151988100 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 10 monclient(hunting): get_auth_request con 0x7f0d340a5b00 auth_method 0 2024-09-13T11:40:17.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-09-13T11:40:17.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 1 -- 172.21.15.5:0/2151988100 >> v1:172.21.15.39:6789/0 conn(0x7f0d340a3e70 legacy=0x7f0d340a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 1 -- 172.21.15.5:0/2151988100 >> v1:172.21.15.5:6789/0 conn(0x7f0d3413a720 legacy=0x7f0d3413cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 1 monclient: found mon.noname-b 2024-09-13T11:40:17.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.311+0000 7f0d337fe640 1 -- 172.21.15.5:0/2151988100 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d34072ed0 con 0x7f0d340a5b00 2024-09-13T11:40:17.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d337fe640 1 --2- 172.21.15.5:0/2151988100 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a5b00 0x7f0d34073e50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0d240098a0 tx=0x7f0d2401c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6f031c4aae9af52 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 1 -- 172.21.15.5:0/2151988100 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d240b0070 con 0x7f0d340a5b00 2024-09-13T11:40:17.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:17.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 10 monclient: dump: 2024-09-13T11:40:17.318 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.318 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.318 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.318 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.318 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.319 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.319 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.319 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.319 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.319 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 1 -- 172.21.15.5:0/2151988100 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d24004230 con 0x7f0d340a5b00 2024-09-13T11:40:17.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 5 monclient: authenticate success, global_id 4268 2024-09-13T11:40:17.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:17.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 1 -- 172.21.15.5:0/2151988100 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24003620 con 0x7f0d340a5b00 2024-09-13T11:40:17.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 4 set_mon_vals no callback set 2024-09-13T11:40:17.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:17.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d327fc640 10 monclient: dump: 2024-09-13T11:40:17.321 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.321 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.321 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.321 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.321 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.321 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.321 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.321 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.321 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.321 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient: shutdown 2024-09-13T11:40:17.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 -- 172.21.15.5:0/2151988100 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a5b00 msgr2=0x7f0d34073e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 --2- 172.21.15.5:0/2151988100 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a5b00 0x7f0d34073e50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0d240098a0 tx=0x7f0d2401c440 comp rx=0 tx=0).stop 2024-09-13T11:40:17.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 -- 172.21.15.5:0/2151988100 shutdown_connections 2024-09-13T11:40:17.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 --2- 172.21.15.5:0/2151988100 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a5b00 0x7f0d34073e50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 -- 172.21.15.5:0/2151988100 >> 172.21.15.5:0/2151988100 conn(0x7f0d340a6da0 msgr2=0x7f0d340a32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 -- 172.21.15.5:0/2151988100 shutdown_connections 2024-09-13T11:40:17.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 -- 172.21.15.5:0/2151988100 wait complete. 2024-09-13T11:40:17.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient: monmap: 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 Processor -- start 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 -- start start 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient: init 2024-09-13T11:40:17.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a3e70 0x7f0d3413ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient(hunting): picked mon.noname-c con 0x7f0d340a3e70 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d340a5b00 0x7f0d3413bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient(hunting): picked mon.noname-b con 0x7f0d340a5b00 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d3413a720 0x7f0d3413c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient(hunting): picked mon.noname-a con 0x7f0d3413a720 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0d34140e50 con 0x7f0d3413a720 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0d34140cd0 con 0x7f0d340a5b00 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f0d34140fd0 con 0x7f0d340a3e70 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient(hunting): authenticate will time out at 1020.666862s 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d337fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a3e70 0x7f0d3413ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d337fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a3e70 0x7f0d3413ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59308/0 (socket says 172.21.15.5:59308) 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d337fe640 1 -- 172.21.15.5:0/700718665 learned_addr learned my addr 172.21.15.5:0/700718665 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d337fe640 10 monclient(hunting): get_auth_request con 0x7f0d340a3e70 auth_method 0 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d337fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 1 --2- 172.21.15.5:0/700718665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d3413a720 0x7f0d3413c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 10 monclient(hunting): get_auth_request con 0x7f0d3413a720 auth_method 0 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d32ffd640 1 --2- 172.21.15.5:0/700718665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d340a5b00 0x7f0d3413bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d32ffd640 10 monclient(hunting): get_auth_request con 0x7f0d340a5b00 auth_method 0 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 1 -- 172.21.15.5:0/700718665 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a3e70 msgr2=0x7f0d3413ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 1 --2- 172.21.15.5:0/700718665 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a3e70 0x7f0d3413ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 1 -- 172.21.15.5:0/700718665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d340a5b00 msgr2=0x7f0d3413bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 1 --2- 172.21.15.5:0/700718665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d340a5b00 0x7f0d3413bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 1 monclient: found mon.noname-a 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 1 -- 172.21.15.5:0/700718665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d34136a40 con 0x7f0d3413a720 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d337fe640 1 --2- 172.21.15.5:0/700718665 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a3e70 0x7f0d3413ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d32ffd640 1 --2- 172.21.15.5:0/700718665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d340a5b00 0x7f0d3413bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d33fff640 1 --2- 172.21.15.5:0/700718665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d3413a720 0x7f0d3413c4f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00c700 tx=0x7f0d2c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 1 -- 172.21.15.5:0/700718665 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c003b80 con 0x7f0d3413a720 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:17.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 10 monclient: dump: 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 1 -- 172.21.15.5:0/700718665 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d2c003d20 con 0x7f0d3413a720 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 5 monclient: authenticate success, global_id 4332 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient: _renew_subs 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 -- 172.21.15.5:0/700718665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d34137ec0 con 0x7f0d3413a720 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 1 -- 172.21.15.5:0/700718665 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c00aa50 con 0x7f0d3413a720 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d1bfff640 10 monclient: dump: 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.336 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient: _renew_subs 2024-09-13T11:40:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d31ffb640 4 set_mon_vals no callback set 2024-09-13T11:40:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.315+0000 7f0d39abc640 1 -- 172.21.15.5:0/700718665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d341381a0 con 0x7f0d3413a720 2024-09-13T11:40:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.319+0000 7f0d39abc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.319+0000 7f0d39abc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.319+0000 7f0d39abc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.319+0000 7f0d39abc640 1 -- 172.21.15.5:0/700718665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d34073e50 con 0x7f0d3413a720 2024-09-13T11:40:17.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.319+0000 7f0d1bfff640 1 -- 172.21.15.5:0/700718665 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f0d2c00abf0 con 0x7f0d3413a720 2024-09-13T11:40:17.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.319+0000 7f0d1bfff640 1 --2- 172.21.15.5:0/700718665 >> v2:172.21.15.39:6812/1340838781 conn(0x7f0d2003b2d0 0x7f0d2003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.319+0000 7f0d1bfff640 1 -- 172.21.15.5:0/700718665 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f0d2c04eaa0 con 0x7f0d3413a720 2024-09-13T11:40:17.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.319+0000 7f0d337fe640 1 --2- 172.21.15.5:0/700718665 >> v2:172.21.15.39:6812/1340838781 conn(0x7f0d2003b2d0 0x7f0d2003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.319+0000 7f0d337fe640 10 monclient: get_auth_request con 0x7f0d2003b2d0 auth_method 0 2024-09-13T11:40:17.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.319+0000 7f0d337fe640 1 --2- 172.21.15.5:0/700718665 >> v2:172.21.15.39:6812/1340838781 conn(0x7f0d2003b2d0 0x7f0d2003d790 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0d240098a0 tx=0x7f0d24009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.323+0000 7f0d1bfff640 1 -- 172.21.15.5:0/700718665 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f0d34136a40 con 0x7f0d3413a720 2024-09-13T11:40:17.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.323+0000 7f0d1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.323+0000 7f0d1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:17.338 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T11:40:17.339 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T11:40:17.339 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T11:40:17.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient: get_monmap_and_config 2024-09-13T11:40:17.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient: monmap: 2024-09-13T11:40:17.351 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.351 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.351 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:17.353423+0000 2024-09-13T11:40:17.351 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:17.353423+0000 2024-09-13T11:40:17.352 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.352 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.352 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:17.352 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:17.352 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:17.352 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:17.352 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:17.352 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 1 Processor -- start 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 1 -- start start 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient: init 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbb600a2970 0x7fbb6009cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient(hunting): picked mon.noname-c con 0x7fbb600a2970 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbb600a2d50 0x7fbb600a3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient(hunting): picked mon.noname-a con 0x7fbb600a2d50 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb60136740 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb6009c1d0 con 0x7fbb60136740 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fbb6009bf80 con 0x7fbb600a2d50 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fbb6009c4b0 con 0x7fbb600a2970 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb65116640 10 monclient(hunting): authenticate will time out at 1020.702554s 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbb600a2d50 0x7fbb600a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbb600a2d50 0x7fbb600a3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57416/0 (socket says 172.21.15.5:57416) 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 1 -- 172.21.15.5:0/4235538113 learned_addr learned my addr 172.21.15.5:0/4235538113 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 10 monclient(hunting): get_auth_request con 0x7fbb600a2d50 auth_method 0 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb5ed9d640 1 --2- 172.21.15.5:0/4235538113 >> v2:172.21.15.39:3300/0 conn(0x7fbb600a2970 0x7fbb6009cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb5ed9d640 10 monclient(hunting): get_auth_request con 0x7fbb600a2970 auth_method 0 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2024-09-13T11:40:17.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 1 -- 172.21.15.5:0/4235538113 >> v2:172.21.15.39:3300/0 conn(0x7fbb600a2970 msgr2=0x7fbb6009cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:17.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 1 --2- 172.21.15.5:0/4235538113 >> v2:172.21.15.39:3300/0 conn(0x7fbb600a2970 0x7fbb6009cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 1 -- 172.21.15.5:0/4235538113 >> v1:172.21.15.39:6789/0 conn(0x7fbb60136740 legacy=0x7fbb60136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 1 monclient: found mon.noname-a 2024-09-13T11:40:17.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.351+0000 7fbb57fff640 1 -- 172.21.15.5:0/4235538113 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6009d0f0 con 0x7fbb600a2d50 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb57fff640 1 --2- 172.21.15.5:0/4235538113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb600a2d50 0x7fbb600a3180 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb60050ec0 tx=0x7fbb4801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12748a2a7864045e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb5e59c640 1 -- 172.21.15.5:0/4235538113 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480ab600 con 0x7fbb600a2d50 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb5e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb5e59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb5e59c640 10 monclient: dump: 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb5e59c640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb5e59c640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb5e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb65116640 5 monclient: authenticate success, global_id 4335 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb65116640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb5e59c640 1 -- 172.21.15.5:0/4235538113 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb480b0070 con 0x7fbb600a2d50 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb5e59c640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb65116640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb65116640 4 set_mon_vals no callback set 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb5e59c640 1 -- 172.21.15.5:0/4235538113 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480ab9c0 con 0x7fbb600a2d50 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb5e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.355+0000 7fbb5e59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb5e59c640 10 monclient: dump: 2024-09-13T11:40:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.361 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient: shutdown 2024-09-13T11:40:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 -- 172.21.15.5:0/4235538113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb600a2d50 msgr2=0x7fbb600a3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 --2- 172.21.15.5:0/4235538113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb600a2d50 0x7fbb600a3180 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb60050ec0 tx=0x7fbb4801c3e0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 -- 172.21.15.5:0/4235538113 shutdown_connections 2024-09-13T11:40:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 --2- 172.21.15.5:0/4235538113 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb600a2d50 0x7fbb600a3180 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 --2- 172.21.15.5:0/4235538113 >> v2:172.21.15.39:3300/0 conn(0x7fbb600a2970 0x7fbb6009cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 -- 172.21.15.5:0/4235538113 >> 172.21.15.5:0/4235538113 conn(0x7fbb600a3c30 msgr2=0x7fbb600a4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 -- 172.21.15.5:0/4235538113 shutdown_connections 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 -- 172.21.15.5:0/4235538113 wait complete. 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient: monmap: 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 Processor -- start 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 -- start start 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient: init 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb600a2970 0x7fbb60135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient(hunting): picked mon.noname-c con 0x7fbb600a2970 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb60136740 0x7fbb60135970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient(hunting): picked mon.noname-a con 0x7fbb60136740 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb6011d5b0 0x7fbb6011da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient(hunting): picked mon.noname-b con 0x7fbb6011d5b0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fbb60142720 con 0x7fbb60136740 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbb601425a0 con 0x7fbb6011d5b0 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fbb601428a0 con 0x7fbb600a2970 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb65116640 10 monclient(hunting): authenticate will time out at 1020.711436s 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb5ed9d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb600a2970 0x7fbb60135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb5ed9d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb600a2970 0x7fbb60135430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59320/0 (socket says 172.21.15.5:59320) 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb5ed9d640 1 -- 172.21.15.5:0/592473746 learned_addr learned my addr 172.21.15.5:0/592473746 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb5ed9d640 10 monclient(hunting): get_auth_request con 0x7fbb600a2970 auth_method 0 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb57fff640 1 --2- 172.21.15.5:0/592473746 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb60136740 0x7fbb60135970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb57fff640 10 monclient(hunting): get_auth_request con 0x7fbb60136740 auth_method 0 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb57fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb5f59e640 1 --2- 172.21.15.5:0/592473746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb6011d5b0 0x7fbb6011da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.359+0000 7fbb57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5f59e640 10 monclient(hunting): get_auth_request con 0x7fbb6011d5b0 auth_method 0 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5f59e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5ed9d640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5ed9d640 1 -- 172.21.15.5:0/592473746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb6011d5b0 msgr2=0x7fbb6011da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5ed9d640 1 --2- 172.21.15.5:0/592473746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb6011d5b0 0x7fbb6011da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5ed9d640 1 -- 172.21.15.5:0/592473746 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb60136740 msgr2=0x7fbb60135970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5ed9d640 1 --2- 172.21.15.5:0/592473746 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb60136740 0x7fbb60135970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5ed9d640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5ed9d640 1 monclient: found mon.noname-c 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5ed9d640 1 -- 172.21.15.5:0/592473746 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6011ecf0 con 0x7fbb600a2970 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb57fff640 1 --2- 172.21.15.5:0/592473746 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb60136740 0x7fbb60135970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5ed9d640 1 --2- 172.21.15.5:0/592473746 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb600a2970 0x7fbb60135430 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000d800 tx=0x7fbb5000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 1 -- 172.21.15.5:0/592473746 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5000b6c0 con 0x7fbb600a2970 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 10 monclient: dump: 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.369 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 1 -- 172.21.15.5:0/592473746 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb50019070 con 0x7fbb600a2970 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 1 -- 172.21.15.5:0/592473746 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5000ba60 con 0x7fbb600a2970 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 10 monclient: dump: 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb65116640 5 monclient: authenticate success, global_id 4274 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb5dd9b640 4 set_mon_vals no callback set 2024-09-13T11:40:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb65116640 10 monclient: _renew_subs 2024-09-13T11:40:17.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb65116640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb65116640 1 -- 172.21.15.5:0/592473746 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb60135eb0 con 0x7fbb600a2970 2024-09-13T11:40:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb65116640 10 monclient: _renew_subs 2024-09-13T11:40:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb65116640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb65116640 1 -- 172.21.15.5:0/592473746 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb600a2460 con 0x7fbb600a2970 2024-09-13T11:40:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 1 -- 172.21.15.5:0/592473746 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fbb5000bc00 con 0x7fbb600a2970 2024-09-13T11:40:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb65116640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb65116640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb65116640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb65116640 1 -- 172.21.15.5:0/592473746 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb600a3180 con 0x7fbb600a2970 2024-09-13T11:40:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 1 --2- 172.21.15.5:0/592473746 >> v2:172.21.15.39:6812/1340838781 conn(0x7fbb4003b2d0 0x7fbb4003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb577fe640 1 -- 172.21.15.5:0/592473746 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fbb5004f190 con 0x7fbb600a2970 2024-09-13T11:40:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb57fff640 1 --2- 172.21.15.5:0/592473746 >> v2:172.21.15.39:6812/1340838781 conn(0x7fbb4003b2d0 0x7fbb4003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.363+0000 7fbb57fff640 10 monclient: get_auth_request con 0x7fbb4003b2d0 auth_method 0 2024-09-13T11:40:17.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.367+0000 7fbb57fff640 1 --2- 172.21.15.5:0/592473746 >> v2:172.21.15.39:6812/1340838781 conn(0x7fbb4003b2d0 0x7fbb4003d790 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbb480af000 tx=0x7fbb480a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.371+0000 7fbb577fe640 1 -- 172.21.15.5:0/592473746 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fbb60135eb0 con 0x7fbb600a2970 2024-09-13T11:40:17.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.371+0000 7fbb577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.371+0000 7fbb577fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient: get_monmap_and_config 2024-09-13T11:40:17.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient: monmap: 2024-09-13T11:40:17.424 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.424 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.424 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:17.425510+0000 2024-09-13T11:40:17.424 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:17.425510+0000 2024-09-13T11:40:17.424 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.424 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.424 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:17.424 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:17.424 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:17.425 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:17.425 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:17.425 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 1 Processor -- start 2024-09-13T11:40:17.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 1 -- start start 2024-09-13T11:40:17.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient: init 2024-09-13T11:40:17.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-13T11:40:17.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa27c137930 0x7fa27c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient(hunting): picked mon.noname-a con 0x7fa27c137930 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:17.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa27c137d10 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:17.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c13cd80 0x7fa27c13f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient(hunting): picked mon.noname-b con 0x7fa27c13cd80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:17.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa27c0a5a80 con 0x7fa27c137d10 2024-09-13T11:40:17.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa27c0a5fe0 con 0x7fa27c137930 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa27c09c540 con 0x7fa27c13cd80 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c13cd80 0x7fa27c13f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c13cd80 0x7fa27c13f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59336/0 (socket says 172.21.15.5:59336) 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 1 -- 172.21.15.5:0/2828573404 learned_addr learned my addr 172.21.15.5:0/2828573404 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa28216b640 10 monclient(hunting): authenticate will time out at 1020.774245s 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 10 monclient(hunting): get_auth_request con 0x7fa27c13cd80 auth_method 0 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27b7fe640 1 --2- 172.21.15.5:0/2828573404 >> v2:172.21.15.5:3300/0 conn(0x7fa27c137930 0x7fa27c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27b7fe640 10 monclient(hunting): get_auth_request con 0x7fa27c137930 auth_method 0 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 1 -- 172.21.15.5:0/2828573404 >> v2:172.21.15.5:3300/0 conn(0x7fa27c137930 msgr2=0x7fa27c09cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 1 --2- 172.21.15.5:0/2828573404 >> v2:172.21.15.5:3300/0 conn(0x7fa27c137930 0x7fa27c09cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 1 -- 172.21.15.5:0/2828573404 >> v1:172.21.15.5:6789/0 conn(0x7fa27c137d10 legacy=0x7fa27c1380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 1 monclient: found mon.noname-b 2024-09-13T11:40:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.423+0000 7fa27bfff640 1 -- 172.21.15.5:0/2828573404 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27c09bef0 con 0x7fa27c13cd80 2024-09-13T11:40:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27bfff640 1 --2- 172.21.15.5:0/2828573404 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c13cd80 0x7fa27c13f170 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa274009e50 tx=0x7fa27401c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cc8c5e48f4fcf90 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 1 -- 172.21.15.5:0/2828573404 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2740a4a70 con 0x7fa27c13cd80 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 10 monclient: dump: 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.431 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 1 -- 172.21.15.5:0/2828573404 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2740a4c10 con 0x7fa27c13cd80 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 5 monclient: authenticate success, global_id 4277 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 1 -- 172.21.15.5:0/2828573404 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2740ac3f0 con 0x7fa27c13cd80 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 4 set_mon_vals no callback set 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa27a7fc640 10 monclient: dump: 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 10 monclient: shutdown 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 1 -- 172.21.15.5:0/2828573404 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c13cd80 msgr2=0x7fa27c13f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 1 --2- 172.21.15.5:0/2828573404 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c13cd80 0x7fa27c13f170 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa274009e50 tx=0x7fa27401c4f0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 1 -- 172.21.15.5:0/2828573404 shutdown_connections 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 1 --2- 172.21.15.5:0/2828573404 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c13cd80 0x7fa27c13f170 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 1 --2- 172.21.15.5:0/2828573404 >> v2:172.21.15.5:3300/0 conn(0x7fa27c137930 0x7fa27c09cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 1 -- 172.21.15.5:0/2828573404 >> 172.21.15.5:0/2828573404 conn(0x7fa27c0a6c70 msgr2=0x7fa27c0a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 1 -- 172.21.15.5:0/2828573404 shutdown_connections 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 1 -- 172.21.15.5:0/2828573404 wait complete. 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 10 monclient: monmap: 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:17.435 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.427+0000 7fa28216b640 1 Processor -- start 2024-09-13T11:40:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 1 -- start start 2024-09-13T11:40:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient: init 2024-09-13T11:40:17.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:40:17.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa27c137930 0x7fa27c13bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient(hunting): picked mon.noname-b con 0x7fa27c137930 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:17.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c137d10 0x7fa27c1d0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient(hunting): picked mon.noname-c con 0x7fa27c137d10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:17.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27c13cd80 0x7fa27c139810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient(hunting): picked mon.noname-a con 0x7fa27c13cd80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:17.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27affd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c137d10 0x7fa27c1d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27affd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c137d10 0x7fa27c1d0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59338/0 (socket says 172.21.15.5:59338) 2024-09-13T11:40:17.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27affd640 1 -- 172.21.15.5:0/3301208461 learned_addr learned my addr 172.21.15.5:0/3301208461 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27b7fe640 1 --2- 172.21.15.5:0/3301208461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa27c137930 0x7fa27c13bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 1 -- 172.21.15.5:0/3301208461 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa27c142b60 con 0x7fa27c13cd80 2024-09-13T11:40:17.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 1 -- 172.21.15.5:0/3301208461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa27c1429e0 con 0x7fa27c137930 2024-09-13T11:40:17.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 1 -- 172.21.15.5:0/3301208461 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa27c142ce0 con 0x7fa27c137d10 2024-09-13T11:40:17.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient(hunting): authenticate will time out at 1020.781400s 2024-09-13T11:40:17.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27affd640 10 monclient(hunting): get_auth_request con 0x7fa27c137d10 auth_method 0 2024-09-13T11:40:17.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 1 --2- 172.21.15.5:0/3301208461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27c13cd80 0x7fa27c139810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27b7fe640 10 monclient(hunting): get_auth_request con 0x7fa27c137930 auth_method 0 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 10 monclient(hunting): get_auth_request con 0x7fa27c13cd80 auth_method 0 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 1 -- 172.21.15.5:0/3301208461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c137d10 msgr2=0x7fa27c1d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 1 --2- 172.21.15.5:0/3301208461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c137d10 0x7fa27c1d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 1 -- 172.21.15.5:0/3301208461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa27c137930 msgr2=0x7fa27c13bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 1 --2- 172.21.15.5:0/3301208461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa27c137930 0x7fa27c13bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 1 monclient: found mon.noname-a 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 1 -- 172.21.15.5:0/3301208461 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27c0a9080 con 0x7fa27c13cd80 2024-09-13T11:40:17.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27b7fe640 1 --2- 172.21.15.5:0/3301208461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa27c137930 0x7fa27c13bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27affd640 1 --2- 172.21.15.5:0/3301208461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c137d10 0x7fa27c1d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa27bfff640 1 --2- 172.21.15.5:0/3301208461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27c13cd80 0x7fa27c139810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa27c1388a0 tx=0x7fa2740195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 1 -- 172.21.15.5:0/3301208461 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2740bb070 con 0x7fa27c13cd80 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 10 monclient: dump: 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.442 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 1 -- 172.21.15.5:0/3301208461 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2740b7660 con 0x7fa27c13cd80 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 5 monclient: authenticate success, global_id 4347 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient: _renew_subs 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 1 -- 172.21.15.5:0/3301208461 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa27c120d40 con 0x7fa27c13cd80 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 1 -- 172.21.15.5:0/3301208461 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2740a4a70 con 0x7fa27c13cd80 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:17.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa263fff640 10 monclient: dump: 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient: _renew_subs 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa28216b640 1 -- 172.21.15.5:0/3301208461 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa27c121020 con 0x7fa27c13cd80 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.431+0000 7fa279ffb640 4 set_mon_vals no callback set 2024-09-13T11:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.435+0000 7fa263fff640 1 -- 172.21.15.5:0/3301208461 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fa2740b7a50 con 0x7fa27c13cd80 2024-09-13T11:40:17.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.435+0000 7fa263fff640 1 --2- 172.21.15.5:0/3301208461 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa26803b2d0 0x7fa26803d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.435+0000 7fa263fff640 1 -- 172.21.15.5:0/3301208461 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fa2740e9540 con 0x7fa27c13cd80 2024-09-13T11:40:17.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.435+0000 7fa27b7fe640 1 --2- 172.21.15.5:0/3301208461 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa26803b2d0 0x7fa26803d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.435+0000 7fa27b7fe640 10 monclient: get_auth_request con 0x7fa26803b2d0 auth_method 0 2024-09-13T11:40:17.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.435+0000 7fa27b7fe640 1 --2- 172.21.15.5:0/3301208461 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa26803b2d0 0x7fa26803d790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009bb0 tx=0x7fa26c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.439+0000 7fa261ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.439+0000 7fa261ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.439+0000 7fa261ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.439+0000 7fa261ffb640 1 -- 172.21.15.5:0/3301208461 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa27c13f0a0 con 0x7fa27c13cd80 2024-09-13T11:40:17.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.447+0000 7fa263fff640 1 -- 172.21.15.5:0/3301208461 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fa2740c4070 con 0x7fa27c13cd80 2024-09-13T11:40:17.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.447+0000 7fa263fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.447+0000 7fa263fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:17.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 10 monclient: get_monmap_and_config 2024-09-13T11:40:17.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 10 monclient: monmap: 2024-09-13T11:40:17.467 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.467 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.467 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:17.468873+0000 2024-09-13T11:40:17.467 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:17.468873+0000 2024-09-13T11:40:17.467 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.467 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.467 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:17.468 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:17.468 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:17.468 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:17.468 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:17.468 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 1 Processor -- start 2024-09-13T11:40:17.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 1 -- start start 2024-09-13T11:40:17.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 10 monclient: init 2024-09-13T11:40:17.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-13T11:40:17.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a4c137930 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:17.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8a4c138420 0x7f8a4c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8a4c138420 0x7f8a4c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8a4c138420 0x7f8a4c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57442/0 (socket says 172.21.15.5:57442) 2024-09-13T11:40:17.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 1 -- 172.21.15.5:0/935424397 learned_addr learned my addr 172.21.15.5:0/935424397 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 10 monclient(hunting): picked mon.noname-a con 0x7f8a4c138420 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:17.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a4c13cd80 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:17.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 1 -- 172.21.15.5:0/935424397 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a4c0a5a80 con 0x7f8a4c13cd80 2024-09-13T11:40:17.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 1 -- 172.21.15.5:0/935424397 --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a4c09bef0 con 0x7f8a4c137930 2024-09-13T11:40:17.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 1 -- 172.21.15.5:0/935424397 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f8a4c0a5fe0 con 0x7f8a4c138420 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a4a7fc640 1 -- 172.21.15.5:0/935424397 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 97083240 0 0) 0x7f8a4c0a5a80 con 0x7f8a4c13cd80 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a5130b640 10 monclient(hunting): authenticate will time out at 1020.819223s 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 10 monclient(hunting): get_auth_request con 0x7f8a4c138420 auth_method 0 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a4a7fc640 10 monclient(hunting): _negotiate 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a4a7fc640 10 monclient(hunting): my global_id is 4350 2024-09-13T11:40:17.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a4a7fc640 1 -- 172.21.15.5:0/935424397 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a38003530 con 0x7f8a4c13cd80 2024-09-13T11:40:17.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a4a7fc640 1 -- 172.21.15.5:0/935424397 <== mon.1 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2374059196 0 0) 0x7f8a4c09bef0 con 0x7f8a4c137930 2024-09-13T11:40:17.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a4a7fc640 10 monclient(hunting): _negotiate 2024-09-13T11:40:17.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a4a7fc640 10 monclient(hunting): my global_id is 4333 2024-09-13T11:40:17.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a4a7fc640 1 -- 172.21.15.5:0/935424397 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a4c0a5a80 con 0x7f8a4c137930 2024-09-13T11:40:17.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2024-09-13T11:40:17.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 1 -- 172.21.15.5:0/935424397 >> v1:172.21.15.39:6789/0 conn(0x7f8a4c137930 legacy=0x7f8a4c137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 1 -- 172.21.15.5:0/935424397 >> v1:172.21.15.5:6789/0 conn(0x7f8a4c13cd80 legacy=0x7f8a4c13f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 1 monclient: found mon.noname-a 2024-09-13T11:40:17.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.467+0000 7f8a43fff640 1 -- 172.21.15.5:0/935424397 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c09c1d0 con 0x7f8a4c138420 2024-09-13T11:40:17.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a43fff640 1 --2- 172.21.15.5:0/935424397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4c138420 0x7f8a4c09cc90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009870 tx=0x7f8a3401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5521f3d5fb985bc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 1 -- 172.21.15.5:0/935424397 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a340b0070 con 0x7f8a4c138420 2024-09-13T11:40:17.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:17.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 10 monclient: dump: 2024-09-13T11:40:17.475 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.475 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 1 -- 172.21.15.5:0/935424397 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a340ab600 con 0x7f8a4c138420 2024-09-13T11:40:17.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 1 -- 172.21.15.5:0/935424397 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a340ab9d0 con 0x7f8a4c138420 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a4a7fc640 10 monclient: dump: 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.477 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.478 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a5130b640 5 monclient: authenticate success, global_id 4353 2024-09-13T11:40:17.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a5130b640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:17.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.471+0000 7f8a5130b640 4 set_mon_vals no callback set 2024-09-13T11:40:17.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient: shutdown 2024-09-13T11:40:17.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 -- 172.21.15.5:0/935424397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4c138420 msgr2=0x7f8a4c09cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 --2- 172.21.15.5:0/935424397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4c138420 0x7f8a4c09cc90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009870 tx=0x7f8a3401c4d0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 -- 172.21.15.5:0/935424397 shutdown_connections 2024-09-13T11:40:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 --2- 172.21.15.5:0/935424397 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4c138420 0x7f8a4c09cc90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 -- 172.21.15.5:0/935424397 >> 172.21.15.5:0/935424397 conn(0x7f8a4c0a6c70 msgr2=0x7f8a4c0a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 -- 172.21.15.5:0/935424397 shutdown_connections 2024-09-13T11:40:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 -- 172.21.15.5:0/935424397 wait complete. 2024-09-13T11:40:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient: monmap: 2024-09-13T11:40:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 Processor -- start 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 -- start start 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient: init 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a4c137930 0x7f8a4c138d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient(hunting): picked mon.noname-b con 0x7f8a4c137930 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a4c138420 0x7f8a4c1392a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a4c138420 0x7f8a4c1392a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a4c138420 0x7f8a4c1392a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59340/0 (socket says 172.21.15.5:59340) 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a43fff640 1 -- 172.21.15.5:0/1181545168 learned_addr learned my addr 172.21.15.5:0/1181545168 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient(hunting): picked mon.noname-c con 0x7f8a4c138420 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 --2- 172.21.15.5:0/1181545168 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4c13cd80 0x7f8a4c1397e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a4affd640 1 --2- 172.21.15.5:0/1181545168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a4c137930 0x7f8a4c138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a4b7fe640 1 --2- 172.21.15.5:0/1181545168 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4c13cd80 0x7f8a4c1397e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient(hunting): picked mon.noname-a con 0x7f8a4c13cd80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c1432c0 con 0x7f8a4c13cd80 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c09c550 con 0x7f8a4c137930 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f8a4c143440 con 0x7f8a4c138420 2024-09-13T11:40:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a5130b640 10 monclient(hunting): authenticate will time out at 1020.827885s 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a43fff640 10 monclient(hunting): get_auth_request con 0x7f8a4c138420 auth_method 0 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a43fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a4affd640 10 monclient(hunting): get_auth_request con 0x7f8a4c137930 auth_method 0 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a4affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.475+0000 7f8a4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 10 monclient(hunting): get_auth_request con 0x7f8a4c13cd80 auth_method 0 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 1 -- 172.21.15.5:0/1181545168 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a4c138420 msgr2=0x7f8a4c1392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 1 --2- 172.21.15.5:0/1181545168 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a4c138420 0x7f8a4c1392a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 1 -- 172.21.15.5:0/1181545168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a4c137930 msgr2=0x7f8a4c138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 1 --2- 172.21.15.5:0/1181545168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a4c137930 0x7f8a4c138d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 1 monclient: found mon.noname-a 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 1 -- 172.21.15.5:0/1181545168 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c13b470 con 0x7f8a4c13cd80 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a4b7fe640 1 --2- 172.21.15.5:0/1181545168 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4c13cd80 0x7f8a4c1397e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400c7b0 tx=0x7f8a4400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 1 -- 172.21.15.5:0/1181545168 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4400ce60 con 0x7f8a4c13cd80 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 10 monclient: dump: 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 1 -- 172.21.15.5:0/1181545168 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a44003b80 con 0x7f8a4c13cd80 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a5130b640 5 monclient: authenticate success, global_id 4356 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a5130b640 10 monclient: _renew_subs 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a5130b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4c13c2e0 con 0x7f8a4c13cd80 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 1 -- 172.21.15.5:0/1181545168 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4400a850 con 0x7f8a4c13cd80 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a49ffb640 4 set_mon_vals no callback set 2024-09-13T11:40:17.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 10 monclient: dump: 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a5130b640 10 monclient: _renew_subs 2024-09-13T11:40:17.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a5130b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4c139f80 con 0x7f8a4c13cd80 2024-09-13T11:40:17.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.479+0000 7f8a437fe640 1 -- 172.21.15.5:0/1181545168 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f8a4400a9f0 con 0x7f8a4c13cd80 2024-09-13T11:40:17.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.483+0000 7f8a5130b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.483+0000 7f8a5130b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.483+0000 7f8a5130b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.483+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a4c0ab260 con 0x7f8a4c13cd80 2024-09-13T11:40:17.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.483+0000 7f8a437fe640 1 --2- 172.21.15.5:0/1181545168 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8a2c03b2a0 0x7f8a2c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.483+0000 7f8a437fe640 1 -- 172.21.15.5:0/1181545168 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f8a44052120 con 0x7f8a4c13cd80 2024-09-13T11:40:17.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.483+0000 7f8a4affd640 1 --2- 172.21.15.5:0/1181545168 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8a2c03b2a0 0x7f8a2c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.483+0000 7f8a4affd640 10 monclient: get_auth_request con 0x7f8a2c03b2a0 auth_method 0 2024-09-13T11:40:17.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.483+0000 7f8a4affd640 1 --2- 172.21.15.5:0/1181545168 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8a2c03b2a0 0x7f8a2c03d760 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c001820 tx=0x7f8a3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.491+0000 7f8a437fe640 1 -- 172.21.15.5:0/1181545168 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f8a44012420 con 0x7f8a4c13cd80 2024-09-13T11:40:17.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.491+0000 7f8a437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.491+0000 7f8a437fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:17.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.499+0000 7f1047d75640 10 monclient: get_monmap_and_config 2024-09-13T11:40:17.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.499+0000 7f1047d75640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.499+0000 7f1047d75640 10 monclient: monmap: 2024-09-13T11:40:17.502 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.502 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.502 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:17.503867+0000 2024-09-13T11:40:17.502 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:17.503867+0000 2024-09-13T11:40:17.502 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.502 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.502 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:17.502 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:17.502 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:17.503 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:17.503 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:17.503 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.499+0000 7f1047d75640 1 Processor -- start 2024-09-13T11:40:17.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.499+0000 7f1047d75640 1 -- start start 2024-09-13T11:40:17.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.499+0000 7f1047d75640 10 monclient: init 2024-09-13T11:40:17.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-13T11:40:17.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1040137930 0x7f104009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 10 monclient(hunting): picked mon.noname-c con 0x7f1040137930 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1040137d10 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f104013cd80 0x7f104013f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 10 monclient(hunting): picked mon.noname-a con 0x7f104013cd80 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10400a5a80 con 0x7f1040137d10 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1045b11640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1040137930 0x7f104009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1045310640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1040137d10 0x7f10401380f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36882/0 (socket says 172.21.15.5:36882) 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1045b11640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1040137930 0x7f104009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.5:52980/0 (socket says 172.21.15.5:52980) 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1045310640 1 -- 172.21.15.5:0/2702435342 learned_addr learned my addr 172.21.15.5:0/2702435342 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f104013cd80 0x7f104013f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 1 -- 172.21.15.5:0/2702435342 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f10400a5fe0 con 0x7f104013cd80 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 1 -- 172.21.15.5:0/2702435342 --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f104009c540 con 0x7f1040137930 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1047d75640 10 monclient(hunting): authenticate will time out at 1020.853247s 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1045b11640 10 monclient(hunting): get_auth_request con 0x7f1040137930 auth_method 0 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1045b11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1045b11640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1045b11640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 10 monclient(hunting): get_auth_request con 0x7f104013cd80 auth_method 0 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1045b11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1045b11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1045b11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 10 monclient(hunting): handle_auth_done global_id 4359 payload 274 2024-09-13T11:40:17.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 1 -- 172.21.15.5:0/2702435342 >> v2:172.21.15.39:3300/0 conn(0x7f1040137930 msgr2=0x7f104009cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:17.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 1 --2- 172.21.15.5:0/2702435342 >> v2:172.21.15.39:3300/0 conn(0x7f1040137930 0x7f104009cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 1 -- 172.21.15.5:0/2702435342 >> v1:172.21.15.5:6789/0 conn(0x7f1040137d10 legacy=0x7f10401380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:17.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 1 monclient: found mon.noname-a 2024-09-13T11:40:17.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 1 -- 172.21.15.5:0/2702435342 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104009bef0 con 0x7f104013cd80 2024-09-13T11:40:17.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.503+0000 7f1046312640 1 --2- 172.21.15.5:0/2702435342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104013cd80 0x7f104013f170 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f10400a7cb0 tx=0x7f103801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a7450d90b34905 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 1 -- 172.21.15.5:0/2702435342 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10380b2070 con 0x7f104013cd80 2024-09-13T11:40:17.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:17.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 10 monclient: dump: 2024-09-13T11:40:17.510 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.510 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.510 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.510 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.510 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.510 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 5 monclient: authenticate success, global_id 4359 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 1 -- 172.21.15.5:0/2702435342 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10380a57c0 con 0x7f104013cd80 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 1 -- 172.21.15.5:0/2702435342 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10380a5b90 con 0x7f104013cd80 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:17.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 4 set_mon_vals no callback set 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1044b0f640 10 monclient: dump: 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 10 monclient: shutdown 2024-09-13T11:40:17.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 1 -- 172.21.15.5:0/2702435342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104013cd80 msgr2=0x7f104013f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 1 --2- 172.21.15.5:0/2702435342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104013cd80 0x7f104013f170 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f10400a7cb0 tx=0x7f103801c7f0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 1 -- 172.21.15.5:0/2702435342 shutdown_connections 2024-09-13T11:40:17.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 1 --2- 172.21.15.5:0/2702435342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104013cd80 0x7f104013f170 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 1 --2- 172.21.15.5:0/2702435342 >> v2:172.21.15.39:3300/0 conn(0x7f1040137930 0x7f104009cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 1 -- 172.21.15.5:0/2702435342 >> 172.21.15.5:0/2702435342 conn(0x7f10400a6c70 msgr2=0x7f10400a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 1 -- 172.21.15.5:0/2702435342 shutdown_connections 2024-09-13T11:40:17.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 1 -- 172.21.15.5:0/2702435342 wait complete. 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 10 monclient: get_monmap_and_config 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 10 monclient: monmap: 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:17.510332+0000 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:17.510332+0000 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:17.514 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:17.515 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:17.515 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 10 monclient: monmap: 2024-09-13T11:40:17.515 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.515 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:17.515 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:17.515 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:17.515 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.515 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.516 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:17.516 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:17.516 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:17.516 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 1 Processor -- start 2024-09-13T11:40:17.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 1 -- start start 2024-09-13T11:40:17.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 10 monclient: init 2024-09-13T11:40:17.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 1 Processor -- start 2024-09-13T11:40:17.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 1 -- start start 2024-09-13T11:40:17.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 10 monclient: init 2024-09-13T11:40:17.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-13T11:40:17.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3a4137930 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:17.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa3a4138420 0x7fa3a409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a259c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa3a4137930 0x7fa3a4137d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36896/0 (socket says 172.21.15.5:36896) 2024-09-13T11:40:17.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 10 monclient(hunting): picked mon.noname-a con 0x7fa3a4138420 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:17.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3a413cd80 0x7fa3a413f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a1d9b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa3a4138420 0x7fa3a409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a1d9b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa3a4138420 0x7fa3a409cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57472/0 (socket says 172.21.15.5:57472) 2024-09-13T11:40:17.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:17.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a2d9d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3a413cd80 0x7fa3a413f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1040137930 0x7f104013be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a2d9d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3a413cd80 0x7fa3a413f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59354/0 (socket says 172.21.15.5:59354) 2024-09-13T11:40:17.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a8e42640 10 monclient(hunting): picked mon.noname-b con 0x7fa3a413cd80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:17.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7fa3a1d9b640 1 -- 172.21.15.5:0/1563679237 learned_addr learned my addr 172.21.15.5:0/1563679237 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 10 monclient(hunting): picked mon.noname-c con 0x7f1040137930 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:17.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.507+0000 7f1047d75640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1040137d10 0x7f10401cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a8e42640 1 -- 172.21.15.5:0/1563679237 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a40a5a80 con 0x7fa3a4137930 2024-09-13T11:40:17.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a8e42640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a8e42640 1 -- 172.21.15.5:0/1563679237 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa3a40a5fe0 con 0x7fa3a4138420 2024-09-13T11:40:17.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a8e42640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a8e42640 1 -- 172.21.15.5:0/1563679237 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a409c540 con 0x7fa3a413cd80 2024-09-13T11:40:17.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1040137930 0x7f104013be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1040137930 0x7f104013be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:59356/0 (socket says 172.21.15.5:59356) 2024-09-13T11:40:17.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a8e42640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 10 monclient(hunting): picked mon.noname-b con 0x7f1040137d10 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:17.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 1 -- 172.21.15.5:0/4228556317 learned_addr learned my addr 172.21.15.5:0/4228556317 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a8e42640 10 monclient(hunting): authenticate will time out at 1020.860167s 2024-09-13T11:40:17.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 10 monclient(hunting): get_auth_request con 0x7fa3a4138420 auth_method 0 2024-09-13T11:40:17.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045310640 1 --2- 172.21.15.5:0/4228556317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1040137d10 0x7f10401cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a2d9d640 10 monclient(hunting): get_auth_request con 0x7fa3a413cd80 auth_method 0 2024-09-13T11:40:17.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a159a640 1 -- 172.21.15.5:0/1563679237 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1981860079 0 0) 0x7fa3a40a5a80 con 0x7fa3a4137930 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a159a640 10 monclient(hunting): _negotiate 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a159a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a159a640 10 monclient(hunting): my global_id is 4365 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a159a640 1 -- 172.21.15.5:0/1563679237 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa38c003530 con 0x7fa3a4137930 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 1 -- 172.21.15.5:0/1563679237 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3a413cd80 msgr2=0x7fa3a413f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 1 --2- 172.21.15.5:0/1563679237 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3a413cd80 0x7fa3a413f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 1 -- 172.21.15.5:0/1563679237 >> v1:172.21.15.5:6789/0 conn(0x7fa3a4137930 legacy=0x7fa3a4137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 1 monclient: found mon.noname-a 2024-09-13T11:40:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7fa3a1d9b640 1 -- 172.21.15.5:0/1563679237 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a409bef0 con 0x7fa3a4138420 2024-09-13T11:40:17.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 1 --2- 172.21.15.5:0/4228556317 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104013cd80 0x7f1040139830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1046312640 1 --2- 172.21.15.5:0/4228556317 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104013cd80 0x7f1040139830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 10 monclient(hunting): picked mon.noname-a con 0x7f104013cd80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:17.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 1 -- 172.21.15.5:0/4228556317 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1040142350 con 0x7f104013cd80 2024-09-13T11:40:17.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 1 -- 172.21.15.5:0/4228556317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f10401421d0 con 0x7f1040137d10 2024-09-13T11:40:17.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 1 -- 172.21.15.5:0/4228556317 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f10401424d0 con 0x7f1040137930 2024-09-13T11:40:17.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 10 monclient(hunting): authenticate will time out at 1020.861384s 2024-09-13T11:40:17.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 10 monclient(hunting): get_auth_request con 0x7f1040137930 auth_method 0 2024-09-13T11:40:17.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045310640 10 monclient(hunting): get_auth_request con 0x7f1040137d10 auth_method 0 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045310640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045310640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045310640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045310640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045310640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045310640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 1 -- 172.21.15.5:0/4228556317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1040137d10 msgr2=0x7f10401cfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 1 --2- 172.21.15.5:0/4228556317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1040137d10 0x7f10401cfdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 1 -- 172.21.15.5:0/4228556317 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104013cd80 msgr2=0x7f1040139830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 1 --2- 172.21.15.5:0/4228556317 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104013cd80 0x7f1040139830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 1 monclient: found mon.noname-c 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 1 -- 172.21.15.5:0/4228556317 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104013f010 con 0x7f1040137930 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1046312640 10 monclient: get_auth_request con 0x7f104013cd80 auth_method 0 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1046312640 1 --2- 172.21.15.5:0/4228556317 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104013cd80 0x7f1040139830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1045b11640 1 --2- 172.21.15.5:0/4228556317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1040137930 0x7f104013be80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f103c00e930 tx=0x7f103c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 1 -- 172.21.15.5:0/4228556317 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c017070 con 0x7f1040137930 2024-09-13T11:40:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 10 monclient: dump: 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 1 -- 172.21.15.5:0/4228556317 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103c00ca00 con 0x7f1040137930 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 1 -- 172.21.15.5:0/4228556317 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c00cdb0 con 0x7f1040137930 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f10367fc640 10 monclient: dump: 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.534 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1037fff640 4 set_mon_vals no callback set 2024-09-13T11:40:17.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 5 monclient: authenticate success, global_id 4289 2024-09-13T11:40:17.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 10 monclient: _renew_subs 2024-09-13T11:40:17.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 1 -- 172.21.15.5:0/4228556317 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10400a1360 con 0x7f1040137930 2024-09-13T11:40:17.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 10 monclient: _renew_subs 2024-09-13T11:40:17.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.511+0000 7f1047d75640 1 -- 172.21.15.5:0/4228556317 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f10400a9060 con 0x7f1040137930 2024-09-13T11:40:17.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7f10367fc640 1 -- 172.21.15.5:0/4228556317 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f103c01a070 con 0x7f1040137930 2024-09-13T11:40:17.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7f10367fc640 1 --2- 172.21.15.5:0/4228556317 >> v2:172.21.15.39:6812/1340838781 conn(0x7f102403b2a0 0x7f102403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7f10367fc640 1 -- 172.21.15.5:0/4228556317 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f103c04edb0 con 0x7f1040137930 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a1d9b640 1 --2- 172.21.15.5:0/1563679237 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a4138420 0x7fa3a409cc90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa398009870 tx=0x7fa39801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d49ed6b2c0014c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 1 -- 172.21.15.5:0/1563679237 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3980b0070 con 0x7fa3a4138420 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 10 monclient: dump: 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.535 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 1 -- 172.21.15.5:0/1563679237 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3980ab600 con 0x7fa3a4138420 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 1 -- 172.21.15.5:0/1563679237 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3980ab9d0 con 0x7fa3a4138420 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a159a640 10 monclient: dump: 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.537 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.537 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.537 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.537 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.537 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.537 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.537 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a8e42640 5 monclient: authenticate success, global_id 4368 2024-09-13T11:40:17.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a8e42640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:17.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7fa3a8e42640 4 set_mon_vals no callback set 2024-09-13T11:40:17.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7f1023fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7f1023fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7f1023fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7f1023fff640 1 -- 172.21.15.5:0/4228556317 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f101c005090 con 0x7f1040137930 2024-09-13T11:40:17.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7f1045310640 1 --2- 172.21.15.5:0/4228556317 >> v2:172.21.15.39:6812/1340838781 conn(0x7f102403b2a0 0x7f102403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7f1045310640 10 monclient: get_auth_request con 0x7f102403b2a0 auth_method 0 2024-09-13T11:40:17.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.515+0000 7f1045310640 1 --2- 172.21.15.5:0/4228556317 >> v2:172.21.15.39:6812/1340838781 conn(0x7f102403b2a0 0x7f102403d760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f10400a0fa0 tx=0x7f1030009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 10 monclient: shutdown 2024-09-13T11:40:17.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 1 -- 172.21.15.5:0/1563679237 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a4138420 msgr2=0x7fa3a409cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 1 --2- 172.21.15.5:0/1563679237 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a4138420 0x7fa3a409cc90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa398009870 tx=0x7fa39801c440 comp rx=0 tx=0).stop 2024-09-13T11:40:17.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 1 -- 172.21.15.5:0/1563679237 shutdown_connections 2024-09-13T11:40:17.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 1 --2- 172.21.15.5:0/1563679237 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3a413cd80 0x7fa3a413f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 1 --2- 172.21.15.5:0/1563679237 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a4138420 0x7fa3a409cc90 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa398009870 tx=0x7fa39801c440 comp rx=0 tx=0).stop 2024-09-13T11:40:17.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 1 -- 172.21.15.5:0/1563679237 >> 172.21.15.5:0/1563679237 conn(0x7fa3a40a6c70 msgr2=0x7fa3a40a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 1 -- 172.21.15.5:0/1563679237 shutdown_connections 2024-09-13T11:40:17.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 1 -- 172.21.15.5:0/1563679237 wait complete. 2024-09-13T11:40:17.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7f10367fc640 1 -- 172.21.15.5:0/4228556317 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f10400a1360 con 0x7f1040137930 2024-09-13T11:40:17.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7f10367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7f10367fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:17.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 10 monclient: monmap: 2024-09-13T11:40:17.545 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.545 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:17.545 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 1 Processor -- start 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 1 -- start start 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 10 monclient: init 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.519+0000 7fa3a8e42640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3a4137930 0x7fa3a4136490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 10 monclient(hunting): picked mon.noname-c con 0x7fa3a4137930 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a413cd80 0x7fa3a41369d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a1d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a413cd80 0x7fa3a41369d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a1d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a413cd80 0x7fa3a41369d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57480/0 (socket says 172.21.15.5:57480) 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a1d9b640 1 -- 172.21.15.5:0/2993230288 learned_addr learned my addr 172.21.15.5:0/2993230288 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 10 monclient(hunting): picked mon.noname-a con 0x7fa3a413cd80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 1 --2- 172.21.15.5:0/2993230288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3a4139a10 0x7fa3a4139e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 10 monclient(hunting): picked mon.noname-b con 0x7fa3a4139a10 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a4142340 con 0x7fa3a413cd80 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a41421c0 con 0x7fa3a4139a10 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a41424c0 con 0x7fa3a4137930 2024-09-13T11:40:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 10 monclient(hunting): authenticate will time out at 1020.872845s 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a1d9b640 10 monclient(hunting): get_auth_request con 0x7fa3a413cd80 auth_method 0 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a2d9d640 1 --2- 172.21.15.5:0/2993230288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3a4139a10 0x7fa3a4139e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 1 --2- 172.21.15.5:0/2993230288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3a4137930 0x7fa3a4136490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 10 monclient(hunting): get_auth_request con 0x7fa3a4137930 auth_method 0 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a2d9d640 10 monclient(hunting): get_auth_request con 0x7fa3a4139a10 auth_method 0 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 1 -- 172.21.15.5:0/2993230288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3a4139a10 msgr2=0x7fa3a4139e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 1 --2- 172.21.15.5:0/2993230288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3a4139a10 0x7fa3a4139e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 1 -- 172.21.15.5:0/2993230288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a413cd80 msgr2=0x7fa3a41369d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 1 --2- 172.21.15.5:0/2993230288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a413cd80 0x7fa3a41369d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 1 monclient: found mon.noname-c 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 1 -- 172.21.15.5:0/2993230288 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a4120f90 con 0x7fa3a4137930 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a259c640 1 --2- 172.21.15.5:0/2993230288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3a4137930 0x7fa3a4136490 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa39400c400 tx=0x7fa39400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 1 -- 172.21.15.5:0/2993230288 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3940074e0 con 0x7fa3a4137930 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 10 monclient: dump: 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.550 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 1 -- 172.21.15.5:0/2993230288 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3940047f0 con 0x7fa3a4137930 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a0d99640 4 set_mon_vals no callback set 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 1 -- 172.21.15.5:0/2993230288 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3940076a0 con 0x7fa3a4137930 2024-09-13T11:40:17.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa392ffd640 10 monclient: dump: 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 5 monclient: authenticate success, global_id 4292 2024-09-13T11:40:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 10 monclient: _renew_subs 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.523+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a4121270 con 0x7fa3a4137930 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.527+0000 7fa3a8e42640 10 monclient: _renew_subs 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.527+0000 7fa3a8e42640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.527+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a4120cb0 con 0x7fa3a4137930 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.527+0000 7fa392ffd640 1 -- 172.21.15.5:0/2993230288 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fa394004be0 con 0x7fa3a4137930 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.527+0000 7fa392ffd640 1 --2- 172.21.15.5:0/2993230288 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa38403b2a0 0x7fa38403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.527+0000 7fa3a8e42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.527+0000 7fa3a8e42640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.527+0000 7fa3a8e42640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.527+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a409cc90 con 0x7fa3a4137930 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.527+0000 7fa3a1d9b640 1 --2- 172.21.15.5:0/2993230288 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa38403b2a0 0x7fa38403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.527+0000 7fa3a1d9b640 10 monclient: get_auth_request con 0x7fa38403b2a0 auth_method 0 2024-09-13T11:40:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.527+0000 7fa392ffd640 1 -- 172.21.15.5:0/2993230288 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fa39404cf80 con 0x7fa3a4137930 2024-09-13T11:40:17.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.535+0000 7fa392ffd640 1 -- 172.21.15.5:0/2993230288 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fa394082cf0 con 0x7fa3a4137930 2024-09-13T11:40:17.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.535+0000 7fa392ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.535+0000 7fa392ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:17.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.535+0000 7fa3a1d9b640 1 --2- 172.21.15.5:0/2993230288 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa38403b2a0 0x7fa38403d760 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa39801aaf0 tx=0x7fa3980a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d39abc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T11:40:17.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d39abc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T11:40:17.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d39abc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d39abc640 1 -- 172.21.15.5:0/700718665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0d341364b0 con 0x7f0d3413a720 2024-09-13T11:40:17.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d1bfff640 1 -- 172.21.15.5:0/700718665 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0d2c00ce10 con 0x7f0d3413a720 2024-09-13T11:40:17.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T11:40:17.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d1bfff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:17.573 INFO:teuthology.orchestra.run.smithi005.stdout:51539607557 2024-09-13T11:40:17.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d39abc640 1 -- 172.21.15.5:0/700718665 >> v2:172.21.15.39:6812/1340838781 conn(0x7f0d2003b2d0 msgr2=0x7f0d2003d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d39abc640 1 --2- 172.21.15.5:0/700718665 >> v2:172.21.15.39:6812/1340838781 conn(0x7f0d2003b2d0 0x7f0d2003d790 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0d240098a0 tx=0x7f0d24009040 comp rx=0 tx=0).stop 2024-09-13T11:40:17.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d39abc640 10 monclient: shutdown 2024-09-13T11:40:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d39abc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d39abc640 1 -- 172.21.15.5:0/700718665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d3413a720 msgr2=0x7f0d3413c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.571+0000 7f0d39abc640 1 --2- 172.21.15.5:0/700718665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d3413a720 0x7f0d3413c4f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00c700 tx=0x7f0d2c00cbc0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.575+0000 7f0d39abc640 1 -- 172.21.15.5:0/700718665 shutdown_connections 2024-09-13T11:40:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.575+0000 7f0d39abc640 1 --2- 172.21.15.5:0/700718665 >> v2:172.21.15.39:6812/1340838781 conn(0x7f0d2003b2d0 0x7f0d2003d790 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.575+0000 7f0d39abc640 1 --2- 172.21.15.5:0/700718665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d3413a720 0x7f0d3413c4f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.575+0000 7f0d39abc640 1 --2- 172.21.15.5:0/700718665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d340a5b00 0x7f0d3413bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.575+0000 7f0d39abc640 1 --2- 172.21.15.5:0/700718665 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d340a3e70 0x7f0d3413ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.575+0000 7f0d39abc640 1 -- 172.21.15.5:0/700718665 >> 172.21.15.5:0/700718665 conn(0x7f0d340a6da0 msgr2=0x7f0d3412ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.575+0000 7f0d39abc640 1 -- 172.21.15.5:0/700718665 shutdown_connections 2024-09-13T11:40:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.575+0000 7f0d39abc640 1 -- 172.21.15.5:0/700718665 wait complete. 2024-09-13T11:40:17.590 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.5 2024-09-13T11:40:17.590 DEBUG:teuthology.parallel:result is None 2024-09-13T11:40:17.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.631+0000 7fa261ffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T11:40:17.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.631+0000 7fa261ffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T11:40:17.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.631+0000 7fa261ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.631+0000 7fa261ffb640 1 -- 172.21.15.5:0/3301208461 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa27c136cb0 con 0x7fa27c13cd80 2024-09-13T11:40:17.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa263fff640 1 -- 172.21.15.5:0/3301208461 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa27c0a9080 con 0x7fa27c13cd80 2024-09-13T11:40:17.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa263fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T11:40:17.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa263fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:17.635 INFO:teuthology.orchestra.run.smithi005.stdout:47244640261 2024-09-13T11:40:17.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 1 -- 172.21.15.5:0/3301208461 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa26803b2d0 msgr2=0x7fa26803d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 1 --2- 172.21.15.5:0/3301208461 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa26803b2d0 0x7fa26803d790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009bb0 tx=0x7fa26c008040 comp rx=0 tx=0).stop 2024-09-13T11:40:17.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 10 monclient: shutdown 2024-09-13T11:40:17.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 1 -- 172.21.15.5:0/3301208461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27c13cd80 msgr2=0x7fa27c139810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 1 --2- 172.21.15.5:0/3301208461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27c13cd80 0x7fa27c139810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa27c1388a0 tx=0x7fa2740195b0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 1 -- 172.21.15.5:0/3301208461 shutdown_connections 2024-09-13T11:40:17.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 1 --2- 172.21.15.5:0/3301208461 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa26803b2d0 0x7fa26803d790 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 1 --2- 172.21.15.5:0/3301208461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27c13cd80 0x7fa27c139810 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 1 --2- 172.21.15.5:0/3301208461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa27c137d10 0x7fa27c1d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 1 --2- 172.21.15.5:0/3301208461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa27c137930 0x7fa27c13bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 1 -- 172.21.15.5:0/3301208461 >> 172.21.15.5:0/3301208461 conn(0x7fa27c0a6c70 msgr2=0x7fa27c13df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 1 -- 172.21.15.5:0/3301208461 shutdown_connections 2024-09-13T11:40:17.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.635+0000 7fa261ffb640 1 -- 172.21.15.5:0/3301208461 wait complete. 2024-09-13T11:40:17.652 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.2 2024-09-13T11:40:17.653 DEBUG:teuthology.parallel:result is None 2024-09-13T11:40:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb65116640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T11:40:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb65116640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T11:40:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb65116640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb65116640 1 -- 172.21.15.5:0/592473746 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbb6013f570 con 0x7fbb600a2970 2024-09-13T11:40:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb577fe640 1 -- 172.21.15.5:0/592473746 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbb5001cd70 con 0x7fbb600a2970 2024-09-13T11:40:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T11:40:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb577fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:17.696 INFO:teuthology.orchestra.run.smithi005.stdout:51539607557 2024-09-13T11:40:17.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb65116640 1 -- 172.21.15.5:0/592473746 >> v2:172.21.15.39:6812/1340838781 conn(0x7fbb4003b2d0 msgr2=0x7fbb4003d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb65116640 1 --2- 172.21.15.5:0/592473746 >> v2:172.21.15.39:6812/1340838781 conn(0x7fbb4003b2d0 0x7fbb4003d790 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbb480af000 tx=0x7fbb480a7000 comp rx=0 tx=0).stop 2024-09-13T11:40:17.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb65116640 10 monclient: shutdown 2024-09-13T11:40:17.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb65116640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb65116640 1 -- 172.21.15.5:0/592473746 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb600a2970 msgr2=0x7fbb60135430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.695+0000 7fbb65116640 1 --2- 172.21.15.5:0/592473746 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb600a2970 0x7fbb60135430 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000d800 tx=0x7fbb5000dcc0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.699+0000 7fbb65116640 1 -- 172.21.15.5:0/592473746 shutdown_connections 2024-09-13T11:40:17.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.699+0000 7fbb65116640 1 --2- 172.21.15.5:0/592473746 >> v2:172.21.15.39:6812/1340838781 conn(0x7fbb4003b2d0 0x7fbb4003d790 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.699+0000 7fbb65116640 1 --2- 172.21.15.5:0/592473746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb6011d5b0 0x7fbb6011da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.699+0000 7fbb65116640 1 --2- 172.21.15.5:0/592473746 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb60136740 0x7fbb60135970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.699+0000 7fbb65116640 1 --2- 172.21.15.5:0/592473746 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbb600a2970 0x7fbb60135430 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.699+0000 7fbb65116640 1 -- 172.21.15.5:0/592473746 >> 172.21.15.5:0/592473746 conn(0x7fbb600a3c30 msgr2=0x7fbb600a5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.699+0000 7fbb65116640 1 -- 172.21.15.5:0/592473746 shutdown_connections 2024-09-13T11:40:17.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.699+0000 7fbb65116640 1 -- 172.21.15.5:0/592473746 wait complete. 2024-09-13T11:40:17.719 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.1 2024-09-13T11:40:17.719 DEBUG:teuthology.parallel:result is None 2024-09-13T11:40:17.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.747+0000 7f1023fff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T11:40:17.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.747+0000 7f1023fff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T11:40:17.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.747+0000 7f1023fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.747+0000 7f1023fff640 1 -- 172.21.15.5:0/4228556317 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f101c005650 con 0x7f1040137930 2024-09-13T11:40:17.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.747+0000 7f10367fc640 1 -- 172.21.15.5:0/4228556317 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f104013f010 con 0x7f1040137930 2024-09-13T11:40:17.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.747+0000 7f10367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T11:40:17.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.747+0000 7f10367fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:17.751 INFO:teuthology.orchestra.run.smithi005.stdout:47244640261 2024-09-13T11:40:17.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 1 -- 172.21.15.5:0/4228556317 >> v2:172.21.15.39:6812/1340838781 conn(0x7f102403b2a0 msgr2=0x7f102403d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 1 --2- 172.21.15.5:0/4228556317 >> v2:172.21.15.39:6812/1340838781 conn(0x7f102403b2a0 0x7f102403d760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f10400a0fa0 tx=0x7f1030009040 comp rx=0 tx=0).stop 2024-09-13T11:40:17.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 10 monclient: shutdown 2024-09-13T11:40:17.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 1 -- 172.21.15.5:0/4228556317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1040137930 msgr2=0x7f104013be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 1 --2- 172.21.15.5:0/4228556317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1040137930 0x7f104013be80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f103c00e930 tx=0x7f103c00edf0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 1 -- 172.21.15.5:0/4228556317 shutdown_connections 2024-09-13T11:40:17.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 1 --2- 172.21.15.5:0/4228556317 >> v2:172.21.15.39:6812/1340838781 conn(0x7f102403b2a0 0x7f102403d760 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 1 --2- 172.21.15.5:0/4228556317 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104013cd80 0x7f1040139830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 1 --2- 172.21.15.5:0/4228556317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1040137d10 0x7f10401cfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 1 --2- 172.21.15.5:0/4228556317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1040137930 0x7f104013be80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 1 -- 172.21.15.5:0/4228556317 >> 172.21.15.5:0/4228556317 conn(0x7f10400a6c70 msgr2=0x7f104013dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 1 -- 172.21.15.5:0/4228556317 shutdown_connections 2024-09-13T11:40:17.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.751+0000 7f1023fff640 1 -- 172.21.15.5:0/4228556317 wait complete. 2024-09-13T11:40:17.774 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.0 2024-09-13T11:40:17.774 DEBUG:teuthology.parallel:result is None 2024-09-13T11:40:17.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T11:40:17.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T11:40:17.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa3a406cbe0 con 0x7fa3a4137930 2024-09-13T11:40:17.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa392ffd640 1 -- 172.21.15.5:0/2993230288 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa394010300 con 0x7fa3a4137930 2024-09-13T11:40:17.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa392ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T11:40:17.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa392ffd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:17.776 INFO:teuthology.orchestra.run.smithi005.stdout:51539607557 2024-09-13T11:40:17.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7f8a5130b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T11:40:17.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7f8a5130b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T11:40:17.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7f8a5130b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:17.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f8a4c06cbe0 con 0x7f8a4c13cd80 2024-09-13T11:40:17.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7f8a437fe640 1 -- 172.21.15.5:0/1181545168 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8a44011c80 con 0x7f8a4c13cd80 2024-09-13T11:40:17.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7f8a437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T11:40:17.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7f8a437fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:17.778 INFO:teuthology.orchestra.run.smithi005.stdout:51539607557 2024-09-13T11:40:17.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa38403b2a0 msgr2=0x7fa38403d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 --2- 172.21.15.5:0/2993230288 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa38403b2a0 0x7fa38403d760 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa39801aaf0 tx=0x7fa3980a6000 comp rx=0 tx=0).stop 2024-09-13T11:40:17.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 10 monclient: shutdown 2024-09-13T11:40:17.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3a4137930 msgr2=0x7fa3a4136490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 --2- 172.21.15.5:0/2993230288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3a4137930 0x7fa3a4136490 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa39400c400 tx=0x7fa39400c8c0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 shutdown_connections 2024-09-13T11:40:17.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 --2- 172.21.15.5:0/2993230288 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa38403b2a0 0x7fa38403d760 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 --2- 172.21.15.5:0/2993230288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3a4139a10 0x7fa3a4139e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 --2- 172.21.15.5:0/2993230288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3a413cd80 0x7fa3a41369d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 --2- 172.21.15.5:0/2993230288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa3a4137930 0x7fa3a4136490 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 >> 172.21.15.5:0/2993230288 conn(0x7fa3a40a6c70 msgr2=0x7fa3a413e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 shutdown_connections 2024-09-13T11:40:17.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.775+0000 7fa3a8e42640 1 -- 172.21.15.5:0/2993230288 wait complete. 2024-09-13T11:40:17.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8a2c03b2a0 msgr2=0x7f8a2c03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 1 --2- 172.21.15.5:0/1181545168 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8a2c03b2a0 0x7f8a2c03d760 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c001820 tx=0x7f8a3c009040 comp rx=0 tx=0).stop 2024-09-13T11:40:17.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 10 monclient: shutdown 2024-09-13T11:40:17.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4c13cd80 msgr2=0x7f8a4c1397e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 1 --2- 172.21.15.5:0/1181545168 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4c13cd80 0x7f8a4c1397e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400c7b0 tx=0x7f8a4400cc70 comp rx=0 tx=0).stop 2024-09-13T11:40:17.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 shutdown_connections 2024-09-13T11:40:17.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 1 --2- 172.21.15.5:0/1181545168 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8a2c03b2a0 0x7f8a2c03d760 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 1 --2- 172.21.15.5:0/1181545168 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4c13cd80 0x7f8a4c1397e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 1 --2- 172.21.15.5:0/1181545168 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8a4c138420 0x7f8a4c1392a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 1 --2- 172.21.15.5:0/1181545168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a4c137930 0x7f8a4c138d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 >> 172.21.15.5:0/1181545168 conn(0x7f8a4c0a6c70 msgr2=0x7f8a4c13f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 shutdown_connections 2024-09-13T11:40:17.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.779+0000 7f8a5130b640 1 -- 172.21.15.5:0/1181545168 wait complete. 2024-09-13T11:40:17.795 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.4 2024-09-13T11:40:17.795 DEBUG:teuthology.parallel:result is None 2024-09-13T11:40:17.796 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.3 2024-09-13T11:40:17.796 DEBUG:teuthology.parallel:result is None 2024-09-13T11:40:17.797 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T11:40:17.797 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:40:17.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.947+0000 7fb1151f3640 10 monclient: get_monmap_and_config 2024-09-13T11:40:17.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.947+0000 7fb1151f3640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.947+0000 7fb1151f3640 10 monclient: monmap: 2024-09-13T11:40:17.951 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.951 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.951 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:17.952763+0000 2024-09-13T11:40:17.951 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:17.952763+0000 2024-09-13T11:40:17.951 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.951 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.951 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:17.951 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:17.951 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:17.951 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:17.951 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:17.951 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 1 Processor -- start 2024-09-13T11:40:17.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 1 -- start start 2024-09-13T11:40:17.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 10 monclient: init 2024-09-13T11:40:17.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-13T11:40:17.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb110137930 0x7fb11009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 10 monclient(hunting): picked mon.noname-a con 0x7fb110137930 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:17.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb110137d10 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:17.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb11013cd80 0x7fb11013f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 10 monclient(hunting): picked mon.noname-b con 0x7fb11013cd80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1100a5a80 con 0x7fb110137d10 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fb1100a5fe0 con 0x7fb110137930 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fb11009c540 con 0x7fb11013cd80 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb1151f3640 10 monclient(hunting): authenticate will time out at 1021.301340s 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb11013cd80 0x7fb11013f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10ed9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb110137930 0x7fb11009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb11013cd80 0x7fb11013f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46968/0 (socket says 172.21.15.5:46968) 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 1 -- 172.21.15.5:0/3687580101 learned_addr learned my addr 172.21.15.5:0/3687580101 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 10 monclient(hunting): get_auth_request con 0x7fb11013cd80 auth_method 0 2024-09-13T11:40:17.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10ed9d640 10 monclient(hunting): get_auth_request con 0x7fb110137930 auth_method 0 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-09-13T11:40:17.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 1 -- 172.21.15.5:0/3687580101 >> v2:172.21.15.5:3300/0 conn(0x7fb110137930 msgr2=0x7fb11009cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:17.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 1 --2- 172.21.15.5:0/3687580101 >> v2:172.21.15.5:3300/0 conn(0x7fb110137930 0x7fb11009cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 1 -- 172.21.15.5:0/3687580101 >> v1:172.21.15.5:6789/0 conn(0x7fb110137d10 legacy=0x7fb1101380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 1 monclient: found mon.noname-b 2024-09-13T11:40:17.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 1 -- 172.21.15.5:0/3687580101 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11009bef0 con 0x7fb11013cd80 2024-09-13T11:40:17.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10f59e640 1 --2- 172.21.15.5:0/3687580101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb11013cd80 0x7fb11013f170 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb108009e50 tx=0x7fb10801c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=280950de5713a689 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10dd9b640 1 -- 172.21.15.5:0/3687580101 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1080a56c0 con 0x7fb11013cd80 2024-09-13T11:40:17.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.951+0000 7fb10dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:17.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10dd9b640 10 monclient: dump: 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10dd9b640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10dd9b640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10dd9b640 1 -- 172.21.15.5:0/3687580101 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1080a5860 con 0x7fb11013cd80 2024-09-13T11:40:17.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10dd9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10dd9b640 1 -- 172.21.15.5:0/3687580101 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1080a5c30 con 0x7fb11013cd80 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10dd9b640 10 monclient: dump: 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.963 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 5 monclient: authenticate success, global_id 4295 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 4 set_mon_vals no callback set 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient: shutdown 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 -- 172.21.15.5:0/3687580101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb11013cd80 msgr2=0x7fb11013f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 --2- 172.21.15.5:0/3687580101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb11013cd80 0x7fb11013f170 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb108009e50 tx=0x7fb10801c8f0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 -- 172.21.15.5:0/3687580101 shutdown_connections 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 --2- 172.21.15.5:0/3687580101 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb11013cd80 0x7fb11013f170 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 --2- 172.21.15.5:0/3687580101 >> v2:172.21.15.5:3300/0 conn(0x7fb110137930 0x7fb11009cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 -- 172.21.15.5:0/3687580101 >> 172.21.15.5:0/3687580101 conn(0x7fb1100a6c70 msgr2=0x7fb1100a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 -- 172.21.15.5:0/3687580101 shutdown_connections 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 -- 172.21.15.5:0/3687580101 wait complete. 2024-09-13T11:40:17.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient: build_initial_monmap 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient: monmap: 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 Processor -- start 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 -- start start 2024-09-13T11:40:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient: init 2024-09-13T11:40:17.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:17.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:40:17.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb110137930 0x7fb110136330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient(hunting): picked mon.noname-c con 0x7fb110137930 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:17.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb110137d10 0x7fb110136870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient(hunting): picked mon.noname-a con 0x7fb110137d10 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:17.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb11013cd80 0x7fb110139a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient(hunting): picked mon.noname-b con 0x7fb11013cd80 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:17.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb1101422f0 con 0x7fb110137d10 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb110142170 con 0x7fb11013cd80 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fb110142470 con 0x7fb110137930 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient(hunting): authenticate will time out at 1021.306824s 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb110137d10 0x7fb110136870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb110137930 0x7fb110136330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb110137930 0x7fb110136330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46972/0 (socket says 172.21.15.5:46972) 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 1 -- 172.21.15.5:0/625466631 learned_addr learned my addr 172.21.15.5:0/625466631 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 10 monclient(hunting): get_auth_request con 0x7fb110137930 auth_method 0 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb110137d10 0x7fb110136870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42302/0 (socket says 172.21.15.5:42302) 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10e59c640 10 monclient(hunting): get_auth_request con 0x7fb110137d10 auth_method 0 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10f59e640 1 --2- 172.21.15.5:0/625466631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb11013cd80 0x7fb110139a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-09-13T11:40:17.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 1 -- 172.21.15.5:0/625466631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb11013cd80 msgr2=0x7fb110139a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 1 --2- 172.21.15.5:0/625466631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb11013cd80 0x7fb110139a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 1 -- 172.21.15.5:0/625466631 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb110137d10 msgr2=0x7fb110136870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 1 --2- 172.21.15.5:0/625466631 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb110137d10 0x7fb110136870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 10 monclient: _finish_hunting 0 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 1 monclient: found mon.noname-c 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 1 -- 172.21.15.5:0/625466631 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb110120fd0 con 0x7fb110137930 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10f59e640 10 monclient: get_auth_request con 0x7fb11013cd80 auth_method 0 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10f59e640 1 --2- 172.21.15.5:0/625466631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb11013cd80 0x7fb110139a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10e59c640 1 --2- 172.21.15.5:0/625466631 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb110137d10 0x7fb110136870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10ed9d640 1 --2- 172.21.15.5:0/625466631 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb110137930 0x7fb110136330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb10000e930 tx=0x7fb10000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 1 -- 172.21.15.5:0/625466631 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb100017070 con 0x7fb110137930 2024-09-13T11:40:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 10 monclient: dump: 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:17.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 1 -- 172.21.15.5:0/625466631 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb10000ca00 con 0x7fb110137930 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 1 -- 172.21.15.5:0/625466631 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10000cdb0 con 0x7fb110137930 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 5 monclient: authenticate success, global_id 4298 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient: _renew_subs 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 -- 172.21.15.5:0/625466631 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1101212b0 con 0x7fb110137930 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb10d59a640 4 set_mon_vals no callback set 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb0f77fe640 10 monclient: dump: 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient: _renew_subs 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.955+0000 7fb1151f3640 1 -- 172.21.15.5:0/625466631 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb110120cf0 con 0x7fb110137930 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.959+0000 7fb1151f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.959+0000 7fb1151f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.959+0000 7fb1151f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.959+0000 7fb1151f3640 1 -- 172.21.15.5:0/625466631 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb11009cc90 con 0x7fb110137930 2024-09-13T11:40:17.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.959+0000 7fb0f77fe640 1 -- 172.21.15.5:0/625466631 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fb10001a070 con 0x7fb110137930 2024-09-13T11:40:17.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.959+0000 7fb0f77fe640 1 --2- 172.21.15.5:0/625466631 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb0fc03b2a0 0x7fb0fc03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:17.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.959+0000 7fb0f77fe640 1 -- 172.21.15.5:0/625466631 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fb10004ee20 con 0x7fb110137930 2024-09-13T11:40:17.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.959+0000 7fb10e59c640 1 --2- 172.21.15.5:0/625466631 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb0fc03b2a0 0x7fb0fc03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:17.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.959+0000 7fb10e59c640 10 monclient: get_auth_request con 0x7fb0fc03b2a0 auth_method 0 2024-09-13T11:40:17.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.959+0000 7fb10e59c640 1 --2- 172.21.15.5:0/625466631 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb0fc03b2a0 0x7fb0fc03d760 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb11013bb50 tx=0x7fb104009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:17.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.963+0000 7fb0f77fe640 1 -- 172.21.15.5:0/625466631 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fb100013c20 con 0x7fb110137930 2024-09-13T11:40:17.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.963+0000 7fb0f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:17.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:17.963+0000 7fb0f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:18.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.127+0000 7fb1151f3640 1 -- 172.21.15.5:0/625466631 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb11006cbe0 con 0x7fb0fc03b2a0 2024-09-13T11:40:18.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.127+0000 7fb0f77fe640 1 -- 172.21.15.5:0/625466631 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34153 (secure 0 0 0) 0x7fb11006cbe0 con 0x7fb0fc03b2a0 2024-09-13T11:40:18.431 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:40:18.432 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T11:40:18.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 1 -- 172.21.15.5:0/625466631 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb0fc03b2a0 msgr2=0x7fb0fc03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:18.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 1 --2- 172.21.15.5:0/625466631 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb0fc03b2a0 0x7fb0fc03d760 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb11013bb50 tx=0x7fb104009040 comp rx=0 tx=0).stop 2024-09-13T11:40:18.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 10 monclient: shutdown 2024-09-13T11:40:18.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:18.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 1 -- 172.21.15.5:0/625466631 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb110137930 msgr2=0x7fb110136330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:18.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 1 --2- 172.21.15.5:0/625466631 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb110137930 0x7fb110136330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb10000e930 tx=0x7fb10000edf0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 1 -- 172.21.15.5:0/625466631 shutdown_connections 2024-09-13T11:40:18.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 1 --2- 172.21.15.5:0/625466631 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb0fc03b2a0 0x7fb0fc03d760 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 1 --2- 172.21.15.5:0/625466631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb11013cd80 0x7fb110139a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 1 --2- 172.21.15.5:0/625466631 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb110137d10 0x7fb110136870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 1 --2- 172.21.15.5:0/625466631 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb110137930 0x7fb110136330 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 1 -- 172.21.15.5:0/625466631 >> 172.21.15.5:0/625466631 conn(0x7fb1100a6c70 msgr2=0x7fb11009fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:18.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 1 -- 172.21.15.5:0/625466631 shutdown_connections 2024-09-13T11:40:18.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.131+0000 7fb1151f3640 1 -- 172.21.15.5:0/625466631 wait complete. 2024-09-13T11:40:18.433 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-13T11:40:18.042543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167432,"kb_used_data":1080,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063608,"statfs":{"total":579820584960,"available":579649134592,"internally_reserved":0,"allocated":1105920,"data_stored":1369846,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.942758"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.135514+0000","last_change":"2024-09-13T11:40:14.135514+0000","last_active":"2024-09-13T11:40:14.135514+0000","last_peered":"2024-09-13T11:40:14.135514+0000","last_clean":"2024-09-13T11:40:14.135514+0000","last_became_active":"2024-09-13T11:40:12.461041+0000","last_became_peered":"2024-09-13T11:40:12.461041+0000","last_unstale":"2024-09-13T11:40:14.135514+0000","last_undegraded":"2024-09-13T11:40:14.135514+0000","last_fullsized":"2024-09-13T11:40:14.135514+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:50.409094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126878,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.143285+0000","last_change":"2024-09-13T11:40:14.143285+0000","last_active":"2024-09-13T11:40:14.143285+0000","last_peered":"2024-09-13T11:40:14.143285+0000","last_clean":"2024-09-13T11:40:14.143285+0000","last_became_active":"2024-09-13T11:40:12.106972+0000","last_became_peered":"2024-09-13T11:40:12.106972+0000","last_unstale":"2024-09-13T11:40:14.143285+0000","last_undegraded":"2024-09-13T11:40:14.143285+0000","last_fullsized":"2024-09-13T11:40:14.143285+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:29.565401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001297091,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.135429+0000","last_change":"2024-09-13T11:40:14.135429+0000","last_active":"2024-09-13T11:40:14.135429+0000","last_peered":"2024-09-13T11:40:14.135429+0000","last_clean":"2024-09-13T11:40:14.135429+0000","last_became_active":"2024-09-13T11:40:12.120816+0000","last_became_peered":"2024-09-13T11:40:12.120816+0000","last_unstale":"2024-09-13T11:40:14.135429+0000","last_undegraded":"2024-09-13T11:40:14.135429+0000","last_fullsized":"2024-09-13T11:40:14.135429+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:42.012635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011926969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.143208+0000","last_change":"2024-09-13T11:40:14.143208+0000","last_active":"2024-09-13T11:40:14.143208+0000","last_peered":"2024-09-13T11:40:14.143208+0000","last_clean":"2024-09-13T11:40:14.143208+0000","last_became_active":"2024-09-13T11:40:12.120092+0000","last_became_peered":"2024-09-13T11:40:12.120092+0000","last_unstale":"2024-09-13T11:40:14.143208+0000","last_undegraded":"2024-09-13T11:40:14.143208+0000","last_fullsized":"2024-09-13T11:40:14.143208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:10.230543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001217161,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.140927+0000","last_change":"2024-09-13T11:40:14.139720+0000","last_active":"2024-09-13T11:40:14.140927+0000","last_peered":"2024-09-13T11:40:14.140927+0000","last_clean":"2024-09-13T11:40:14.140927+0000","last_became_active":"2024-09-13T11:40:12.110703+0000","last_became_peered":"2024-09-13T11:40:12.110703+0000","last_unstale":"2024-09-13T11:40:14.140927+0000","last_undegraded":"2024-09-13T11:40:14.140927+0000","last_fullsized":"2024-09-13T11:40:14.140927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:57.779487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021711669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.130605+0000","last_change":"2024-09-13T11:40:14.131038+0000","last_active":"2024-09-13T11:40:14.130605+0000","last_peered":"2024-09-13T11:40:14.130605+0000","last_clean":"2024-09-13T11:40:14.130605+0000","last_became_active":"2024-09-13T11:40:12.102142+0000","last_became_peered":"2024-09-13T11:40:12.102142+0000","last_unstale":"2024-09-13T11:40:14.130605+0000","last_undegraded":"2024-09-13T11:40:14.130605+0000","last_fullsized":"2024-09-13T11:40:14.130605+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:15.679795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046429,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.135976+0000","last_change":"2024-09-13T11:40:14.135976+0000","last_active":"2024-09-13T11:40:14.135976+0000","last_peered":"2024-09-13T11:40:14.135976+0000","last_clean":"2024-09-13T11:40:14.135976+0000","last_became_active":"2024-09-13T11:40:12.105280+0000","last_became_peered":"2024-09-13T11:40:12.105280+0000","last_unstale":"2024-09-13T11:40:14.135976+0000","last_undegraded":"2024-09-13T11:40:14.135976+0000","last_fullsized":"2024-09-13T11:40:14.135976+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:20.791570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001476088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.138397+0000","last_change":"2024-09-13T11:40:14.138397+0000","last_active":"2024-09-13T11:40:14.138397+0000","last_peered":"2024-09-13T11:40:14.138397+0000","last_clean":"2024-09-13T11:40:14.138397+0000","last_became_active":"2024-09-13T11:40:12.113752+0000","last_became_peered":"2024-09-13T11:40:12.113752+0000","last_unstale":"2024-09-13T11:40:14.138397+0000","last_undegraded":"2024-09-13T11:40:14.138397+0000","last_fullsized":"2024-09-13T11:40:14.138397+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:50:52.508777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086960399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.133953+0000","last_change":"2024-09-13T11:40:05.394873+0000","last_active":"2024-09-13T11:40:14.133953+0000","last_peered":"2024-09-13T11:40:14.133953+0000","last_clean":"2024-09-13T11:40:14.133953+0000","last_became_active":"2024-09-13T11:40:05.394176+0000","last_became_peered":"2024-09-13T11:40:05.394176+0000","last_unstale":"2024-09-13T11:40:14.133953+0000","last_undegraded":"2024-09-13T11:40:14.133953+0000","last_fullsized":"2024-09-13T11:40:14.133953+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_clean_scrub_stamp":"2024-09-13T11:40:04.377013+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:00:31.498589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":580920,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":580920,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:40:18.434 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:40:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.559+0000 7f67e81c4640 10 monclient: get_monmap_and_config 2024-09-13T11:40:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.559+0000 7f67e81c4640 10 monclient: build_initial_monmap 2024-09-13T11:40:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 10 monclient: monmap: 2024-09-13T11:40:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:18.564848+0000 2024-09-13T11:40:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:18.564848+0000 2024-09-13T11:40:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:18.564 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:18.564 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:18.564 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:18.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 1 Processor -- start 2024-09-13T11:40:18.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 1 -- start start 2024-09-13T11:40:18.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 10 monclient: init 2024-09-13T11:40:18.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:18.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-13T11:40:18.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f67e0074250 0x7f67e00a0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:18.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 10 monclient(hunting): picked mon.noname-a con 0x7f67e0074250 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:18.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f67e0074630 0x7f67e0074a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:18.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 10 monclient(hunting): picked mon.noname-c con 0x7f67e0074630 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f67e006ea50 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67e009f650 con 0x7f67e006ea50 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f67e00a0570 con 0x7f67e0074250 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f67e009f930 con 0x7f67e0074630 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e81c4640 10 monclient(hunting): authenticate will time out at 1021.913782s 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f67e0074250 0x7f67e00a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f67e0074250 0x7f67e00a0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42312/0 (socket says 172.21.15.5:42312) 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 1 -- 172.21.15.5:0/310723861 learned_addr learned my addr 172.21.15.5:0/310723861 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 10 monclient(hunting): get_auth_request con 0x7f67e0074250 auth_method 0 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:18.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e6761640 1 --1- 172.21.15.5:0/310723861 >> v1:172.21.15.5:6789/0 conn(0x7f67e006ea50 0x7f67e006eec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:56358/0 (socket says 172.21.15.5:56358) 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e575f640 1 --2- 172.21.15.5:0/310723861 >> v2:172.21.15.39:3300/0 conn(0x7f67e0074630 0x7f67e0074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 10 monclient(hunting): handle_auth_done global_id 4383 payload 274 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 1 -- 172.21.15.5:0/310723861 >> v2:172.21.15.39:3300/0 conn(0x7f67e0074630 msgr2=0x7f67e0074a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 1 --2- 172.21.15.5:0/310723861 >> v2:172.21.15.39:3300/0 conn(0x7f67e0074630 0x7f67e0074a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 1 -- 172.21.15.5:0/310723861 >> v1:172.21.15.5:6789/0 conn(0x7f67e006ea50 legacy=0x7f67e006eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 10 monclient: _finish_hunting 0 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 1 monclient: found mon.noname-a 2024-09-13T11:40:18.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:18.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:18.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e5f60640 1 -- 172.21.15.5:0/310723861 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e0074f80 con 0x7f67e0074250 2024-09-13T11:40:18.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e575f640 10 monclient: get_auth_request con 0x7f67e0074630 auth_method 0 2024-09-13T11:40:18.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.563+0000 7f67e575f640 1 --2- 172.21.15.5:0/310723861 >> v2:172.21.15.39:3300/0 conn(0x7f67e0074630 0x7f67e0074a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:18.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 1 --2- 172.21.15.5:0/310723861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67e0074250 0x7f67e00a0030 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f67e006cce0 tx=0x7f67dc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91ecff2c75546ea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:18.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 1 -- 172.21.15.5:0/310723861 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc0b1070 con 0x7f67e0074250 2024-09-13T11:40:18.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 10 monclient: dump: 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 10 monclient: _finish_auth 0 2024-09-13T11:40:18.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 10 monclient: _check_auth_tickets 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 1 -- 172.21.15.5:0/310723861 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67dc0ac600 con 0x7f67e0074250 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 5 monclient: authenticate success, global_id 4383 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 1 -- 172.21.15.5:0/310723861 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc0ac9d0 con 0x7f67e0074250 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 4 set_mon_vals no callback set 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e4f5e640 10 monclient: dump: 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:18.575 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient: shutdown 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 -- 172.21.15.5:0/310723861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67e0074250 msgr2=0x7f67e00a0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 --2- 172.21.15.5:0/310723861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67e0074250 0x7f67e00a0030 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f67e006cce0 tx=0x7f67dc01c3e0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 -- 172.21.15.5:0/310723861 shutdown_connections 2024-09-13T11:40:18.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 --2- 172.21.15.5:0/310723861 >> v2:172.21.15.39:3300/0 conn(0x7f67e0074630 0x7f67e0074a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 --2- 172.21.15.5:0/310723861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67e0074250 0x7f67e00a0030 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 -- 172.21.15.5:0/310723861 >> 172.21.15.5:0/310723861 conn(0x7f67e00a4990 msgr2=0x7f67e00a4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:18.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 -- 172.21.15.5:0/310723861 shutdown_connections 2024-09-13T11:40:18.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 -- 172.21.15.5:0/310723861 wait complete. 2024-09-13T11:40:18.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient: build_initial_monmap 2024-09-13T11:40:18.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient: monmap: 2024-09-13T11:40:18.577 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:18.577 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:18.577 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:18.577 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:18.577 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:18.577 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 Processor -- start 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 -- start start 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient: init 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67e006ea50 0x7f67e0122e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient(hunting): picked mon.noname-c con 0x7f67e006ea50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67e0074250 0x7f67e0124f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient(hunting): picked mon.noname-b con 0x7f67e0074250 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67e0074630 0x7f67e013bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:18.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient(hunting): picked mon.noname-a con 0x7f67e0074630 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f67e0142430 con 0x7f67e0074630 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f67e01422b0 con 0x7f67e0074250 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f67e01425b0 con 0x7f67e006ea50 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e81c4640 10 monclient(hunting): authenticate will time out at 1021.919169s 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e6761640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67e0074630 0x7f67e013bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67e006ea50 0x7f67e0122e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67e006ea50 0x7f67e0122e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46976/0 (socket says 172.21.15.5:46976) 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e6761640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67e0074630 0x7f67e013bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42328/0 (socket says 172.21.15.5:42328) 2024-09-13T11:40:18.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 1 -- 172.21.15.5:0/2896921940 learned_addr learned my addr 172.21.15.5:0/2896921940 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 10 monclient(hunting): get_auth_request con 0x7f67e006ea50 auth_method 0 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e6761640 10 monclient(hunting): get_auth_request con 0x7f67e0074630 auth_method 0 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e6761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e6761640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e6761640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e6761640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e6761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e6761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:18.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e575f640 1 --2- 172.21.15.5:0/2896921940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67e0074250 0x7f67e0124f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e575f640 10 monclient(hunting): get_auth_request con 0x7f67e0074250 auth_method 0 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e575f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e575f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e575f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 1 -- 172.21.15.5:0/2896921940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67e0074250 msgr2=0x7f67e0124f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 1 --2- 172.21.15.5:0/2896921940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67e0074250 0x7f67e0124f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 1 -- 172.21.15.5:0/2896921940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67e0074630 msgr2=0x7f67e013bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 1 --2- 172.21.15.5:0/2896921940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67e0074630 0x7f67e013bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 10 monclient: _finish_hunting 0 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 1 monclient: found mon.noname-c 2024-09-13T11:40:18.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 1 -- 172.21.15.5:0/2896921940 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e0124420 con 0x7f67e006ea50 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e6761640 1 --2- 172.21.15.5:0/2896921940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67e0074630 0x7f67e013bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.567+0000 7f67e5f60640 1 --2- 172.21.15.5:0/2896921940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67e006ea50 0x7f67e0122e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f67dc01a010 tx=0x7f67dc01ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 1 -- 172.21.15.5:0/2896921940 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc0b1040 con 0x7f67e006ea50 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 10 monclient: dump: 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 1 -- 172.21.15.5:0/2896921940 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67dc0b7450 con 0x7f67e006ea50 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 1 -- 172.21.15.5:0/2896921940 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc0b7800 con 0x7f67e006ea50 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 10 monclient: dump: 2024-09-13T11:40:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e81c4640 5 monclient: authenticate success, global_id 4301 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d3fff640 4 set_mon_vals no callback set 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e81c4640 10 monclient: _renew_subs 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e81c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e81c4640 1 -- 172.21.15.5:0/2896921940 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67e009f0f0 con 0x7f67e006ea50 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e81c4640 10 monclient: _renew_subs 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e81c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e81c4640 1 -- 172.21.15.5:0/2896921940 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e01261c0 con 0x7f67e006ea50 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e81c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e81c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e81c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e81c4640 1 -- 172.21.15.5:0/2896921940 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67e00a0030 con 0x7f67e006ea50 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 1 -- 172.21.15.5:0/2896921940 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f67dc0be030 con 0x7f67e006ea50 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 1 --2- 172.21.15.5:0/2896921940 >> v2:172.21.15.39:6812/1340838781 conn(0x7f67c403b2d0 0x7f67c403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67d27fc640 1 -- 172.21.15.5:0/2896921940 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f67dc0e8600 con 0x7f67e006ea50 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e575f640 1 --2- 172.21.15.5:0/2896921940 >> v2:172.21.15.39:6812/1340838781 conn(0x7f67c403b2d0 0x7f67c403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e575f640 10 monclient: get_auth_request con 0x7f67c403b2d0 auth_method 0 2024-09-13T11:40:18.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.571+0000 7f67e575f640 1 --2- 172.21.15.5:0/2896921940 >> v2:172.21.15.39:6812/1340838781 conn(0x7f67c403b2d0 0x7f67c403d790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f67e0124d10 tx=0x7f67d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:18.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.575+0000 7f67d27fc640 1 -- 172.21.15.5:0/2896921940 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f67dc0160f0 con 0x7f67e006ea50 2024-09-13T11:40:18.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.575+0000 7f67d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:18.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.575+0000 7f67d27fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:18.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.735+0000 7f67e81c4640 1 -- 172.21.15.5:0/2896921940 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f67e0126630 con 0x7f67c403b2d0 2024-09-13T11:40:18.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67d27fc640 1 -- 172.21.15.5:0/2896921940 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34153 (secure 0 0 0) 0x7f67d400a670 con 0x7f67c403b2d0 2024-09-13T11:40:18.740 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:40:18.740 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T11:40:18.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 1 -- 172.21.15.5:0/2896921940 >> v2:172.21.15.39:6812/1340838781 conn(0x7f67c403b2d0 msgr2=0x7f67c403d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:18.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 1 --2- 172.21.15.5:0/2896921940 >> v2:172.21.15.39:6812/1340838781 conn(0x7f67c403b2d0 0x7f67c403d790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f67e0124d10 tx=0x7f67d4008040 comp rx=0 tx=0).stop 2024-09-13T11:40:18.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 10 monclient: shutdown 2024-09-13T11:40:18.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:18.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 1 -- 172.21.15.5:0/2896921940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67e006ea50 msgr2=0x7f67e0122e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:18.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 1 --2- 172.21.15.5:0/2896921940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67e006ea50 0x7f67e0122e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f67dc01a010 tx=0x7f67dc01ac20 comp rx=0 tx=0).stop 2024-09-13T11:40:18.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 1 -- 172.21.15.5:0/2896921940 shutdown_connections 2024-09-13T11:40:18.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 1 --2- 172.21.15.5:0/2896921940 >> v2:172.21.15.39:6812/1340838781 conn(0x7f67c403b2d0 0x7f67c403d790 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 1 --2- 172.21.15.5:0/2896921940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f67e0074630 0x7f67e013bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 1 --2- 172.21.15.5:0/2896921940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67e0074250 0x7f67e0124f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 1 --2- 172.21.15.5:0/2896921940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f67e006ea50 0x7f67e0122e70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 1 -- 172.21.15.5:0/2896921940 >> 172.21.15.5:0/2896921940 conn(0x7f67e00a4990 msgr2=0x7f67e013ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:18.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 1 -- 172.21.15.5:0/2896921940 shutdown_connections 2024-09-13T11:40:18.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.739+0000 7f67e81c4640 1 -- 172.21.15.5:0/2896921940 wait complete. 2024-09-13T11:40:18.756 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-13T11:40:18.299826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167432,"kb_used_data":1080,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063608,"statfs":{"total":579820584960,"available":579649134592,"internally_reserved":0,"allocated":1105920,"data_stored":1369846,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001690"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.135514+0000","last_change":"2024-09-13T11:40:14.135514+0000","last_active":"2024-09-13T11:40:14.135514+0000","last_peered":"2024-09-13T11:40:14.135514+0000","last_clean":"2024-09-13T11:40:14.135514+0000","last_became_active":"2024-09-13T11:40:12.461041+0000","last_became_peered":"2024-09-13T11:40:12.461041+0000","last_unstale":"2024-09-13T11:40:14.135514+0000","last_undegraded":"2024-09-13T11:40:14.135514+0000","last_fullsized":"2024-09-13T11:40:14.135514+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:50.409094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126878,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.143285+0000","last_change":"2024-09-13T11:40:14.143285+0000","last_active":"2024-09-13T11:40:14.143285+0000","last_peered":"2024-09-13T11:40:14.143285+0000","last_clean":"2024-09-13T11:40:14.143285+0000","last_became_active":"2024-09-13T11:40:12.106972+0000","last_became_peered":"2024-09-13T11:40:12.106972+0000","last_unstale":"2024-09-13T11:40:14.143285+0000","last_undegraded":"2024-09-13T11:40:14.143285+0000","last_fullsized":"2024-09-13T11:40:14.143285+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:29.565401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001297091,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.135429+0000","last_change":"2024-09-13T11:40:14.135429+0000","last_active":"2024-09-13T11:40:14.135429+0000","last_peered":"2024-09-13T11:40:14.135429+0000","last_clean":"2024-09-13T11:40:14.135429+0000","last_became_active":"2024-09-13T11:40:12.120816+0000","last_became_peered":"2024-09-13T11:40:12.120816+0000","last_unstale":"2024-09-13T11:40:14.135429+0000","last_undegraded":"2024-09-13T11:40:14.135429+0000","last_fullsized":"2024-09-13T11:40:14.135429+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:42.012635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011926969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.143208+0000","last_change":"2024-09-13T11:40:14.143208+0000","last_active":"2024-09-13T11:40:14.143208+0000","last_peered":"2024-09-13T11:40:14.143208+0000","last_clean":"2024-09-13T11:40:14.143208+0000","last_became_active":"2024-09-13T11:40:12.120092+0000","last_became_peered":"2024-09-13T11:40:12.120092+0000","last_unstale":"2024-09-13T11:40:14.143208+0000","last_undegraded":"2024-09-13T11:40:14.143208+0000","last_fullsized":"2024-09-13T11:40:14.143208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:10.230543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001217161,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.140927+0000","last_change":"2024-09-13T11:40:14.139720+0000","last_active":"2024-09-13T11:40:14.140927+0000","last_peered":"2024-09-13T11:40:14.140927+0000","last_clean":"2024-09-13T11:40:14.140927+0000","last_became_active":"2024-09-13T11:40:12.110703+0000","last_became_peered":"2024-09-13T11:40:12.110703+0000","last_unstale":"2024-09-13T11:40:14.140927+0000","last_undegraded":"2024-09-13T11:40:14.140927+0000","last_fullsized":"2024-09-13T11:40:14.140927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:57.779487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021711669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.130605+0000","last_change":"2024-09-13T11:40:14.131038+0000","last_active":"2024-09-13T11:40:14.130605+0000","last_peered":"2024-09-13T11:40:14.130605+0000","last_clean":"2024-09-13T11:40:14.130605+0000","last_became_active":"2024-09-13T11:40:12.102142+0000","last_became_peered":"2024-09-13T11:40:12.102142+0000","last_unstale":"2024-09-13T11:40:14.130605+0000","last_undegraded":"2024-09-13T11:40:14.130605+0000","last_fullsized":"2024-09-13T11:40:14.130605+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:15.679795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046429,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.135976+0000","last_change":"2024-09-13T11:40:14.135976+0000","last_active":"2024-09-13T11:40:14.135976+0000","last_peered":"2024-09-13T11:40:14.135976+0000","last_clean":"2024-09-13T11:40:14.135976+0000","last_became_active":"2024-09-13T11:40:12.105280+0000","last_became_peered":"2024-09-13T11:40:12.105280+0000","last_unstale":"2024-09-13T11:40:14.135976+0000","last_undegraded":"2024-09-13T11:40:14.135976+0000","last_fullsized":"2024-09-13T11:40:14.135976+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:20.791570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001476088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.138397+0000","last_change":"2024-09-13T11:40:14.138397+0000","last_active":"2024-09-13T11:40:14.138397+0000","last_peered":"2024-09-13T11:40:14.138397+0000","last_clean":"2024-09-13T11:40:14.138397+0000","last_became_active":"2024-09-13T11:40:12.113752+0000","last_became_peered":"2024-09-13T11:40:12.113752+0000","last_unstale":"2024-09-13T11:40:14.138397+0000","last_undegraded":"2024-09-13T11:40:14.138397+0000","last_fullsized":"2024-09-13T11:40:14.138397+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:50:52.508777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086960399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.133953+0000","last_change":"2024-09-13T11:40:05.394873+0000","last_active":"2024-09-13T11:40:14.133953+0000","last_peered":"2024-09-13T11:40:14.133953+0000","last_clean":"2024-09-13T11:40:14.133953+0000","last_became_active":"2024-09-13T11:40:05.394176+0000","last_became_peered":"2024-09-13T11:40:05.394176+0000","last_unstale":"2024-09-13T11:40:14.133953+0000","last_undegraded":"2024-09-13T11:40:14.133953+0000","last_fullsized":"2024-09-13T11:40:14.133953+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_clean_scrub_stamp":"2024-09-13T11:40:04.377013+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:00:31.498589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":580920,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":580920,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:40:18.757 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T11:40:18.757 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T11:40:18.757 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T11:40:18.758 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T11:40:18.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 10 monclient: get_monmap_and_config 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 10 monclient: build_initial_monmap 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 10 monclient: monmap: 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:18.877822+0000 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:18.877822+0000 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:18.876 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:18.877 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:18.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 1 Processor -- start 2024-09-13T11:40:18.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 1 -- start start 2024-09-13T11:40:18.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 10 monclient: init 2024-09-13T11:40:18.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:18.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-13T11:40:18.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc0c134810 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:18.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc0c137660 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:18.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbc0c13b170 0x7fbc0c13d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:18.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0c13b170 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc0c09ce60 con 0x7fbc0c137660 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc0c09eeb0 con 0x7fbc0c134810 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fbc0c0a0950 con 0x7fbc0c13b170 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc13938640 10 monclient(hunting): authenticate will time out at 1022.226473s 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbc0c13b170 0x7fbc0c13d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc10ed3640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fbc0c137660 0x7fbc0c139a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:56372/0 (socket says 172.21.15.5:56372) 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc10ed3640 1 -- 172.21.15.5:0/2238460181 learned_addr learned my addr 172.21.15.5:0/2238460181 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 10 monclient(hunting): get_auth_request con 0x7fbc0c13b170 auth_method 0 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:18.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:18.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:18.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:18.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:18.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 10 monclient(hunting): handle_auth_done global_id 4392 payload 274 2024-09-13T11:40:18.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 1 -- 172.21.15.5:0/2238460181 >> v1:172.21.15.39:6789/0 conn(0x7fbc0c134810 legacy=0x7fbc0c134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:18.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 1 -- 172.21.15.5:0/2238460181 >> v1:172.21.15.5:6789/0 conn(0x7fbc0c137660 legacy=0x7fbc0c139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:18.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 10 monclient: _finish_hunting 0 2024-09-13T11:40:18.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 1 monclient: found mon.noname-a 2024-09-13T11:40:18.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:18.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:18.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.875+0000 7fbc11ed5640 1 -- 172.21.15.5:0/2238460181 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c09e1e0 con 0x7fbc0c13b170 2024-09-13T11:40:18.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc11ed5640 1 --2- 172.21.15.5:0/2238460181 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c13b170 0x7fbc0c13d560 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009d90 tx=0x7fbbfc01c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b816dac0e8b125c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:18.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 1 -- 172.21.15.5:0/2238460181 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc0a6700 con 0x7fbc0c13b170 2024-09-13T11:40:18.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 10 monclient: dump: 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 10 monclient: _finish_auth 0 2024-09-13T11:40:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 5 monclient: authenticate success, global_id 4392 2024-09-13T11:40:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 1 -- 172.21.15.5:0/2238460181 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbfc0a68a0 con 0x7fbc0c13b170 2024-09-13T11:40:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 4 set_mon_vals no callback set 2024-09-13T11:40:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 1 -- 172.21.15.5:0/2238460181 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc0a6c70 con 0x7fbc0c13b170 2024-09-13T11:40:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0bfff640 10 monclient: dump: 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient: shutdown 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:18.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 -- 172.21.15.5:0/2238460181 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c13b170 msgr2=0x7fbc0c13d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:18.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 --2- 172.21.15.5:0/2238460181 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c13b170 0x7fbc0c13d560 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009d90 tx=0x7fbbfc01c830 comp rx=0 tx=0).stop 2024-09-13T11:40:18.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 -- 172.21.15.5:0/2238460181 shutdown_connections 2024-09-13T11:40:18.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 --2- 172.21.15.5:0/2238460181 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c13b170 0x7fbc0c13d560 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 -- 172.21.15.5:0/2238460181 >> 172.21.15.5:0/2238460181 conn(0x7fbc0c0a2930 msgr2=0x7fbc0c0a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:18.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 -- 172.21.15.5:0/2238460181 shutdown_connections 2024-09-13T11:40:18.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 -- 172.21.15.5:0/2238460181 wait complete. 2024-09-13T11:40:18.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient: build_initial_monmap 2024-09-13T11:40:18.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient: monmap: 2024-09-13T11:40:18.889 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 Processor -- start 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 -- start start 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient: init 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c134810 0x7fbc0c1cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0c134810 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc0c137660 0x7fbc0c1cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0c137660 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c13b170 0x7fbc0c1ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0c13b170 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c0a16c0 con 0x7fbc0c134810 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c0a1540 con 0x7fbc0c13b170 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fbc0c0a1840 con 0x7fbc0c137660 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient(hunting): authenticate will time out at 1022.230999s 2024-09-13T11:40:18.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc0c137660 0x7fbc0c1cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc0c137660 0x7fbc0c1cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46980/0 (socket says 172.21.15.5:46980) 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 1 -- 172.21.15.5:0/2042638274 learned_addr learned my addr 172.21.15.5:0/2042638274 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 10 monclient(hunting): get_auth_request con 0x7fbc0c137660 auth_method 0 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc116d4640 1 --2- 172.21.15.5:0/2042638274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c134810 0x7fbc0c1cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc116d4640 10 monclient(hunting): get_auth_request con 0x7fbc0c134810 auth_method 0 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc116d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc116d4640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc116d4640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc116d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc116d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc116d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc11ed5640 1 --2- 172.21.15.5:0/2042638274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c13b170 0x7fbc0c1ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 1 -- 172.21.15.5:0/2042638274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c13b170 msgr2=0x7fbc0c1ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 1 --2- 172.21.15.5:0/2042638274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c13b170 0x7fbc0c1ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 1 -- 172.21.15.5:0/2042638274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c134810 msgr2=0x7fbc0c1cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 1 --2- 172.21.15.5:0/2042638274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c134810 0x7fbc0c1cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 10 monclient: _finish_hunting 0 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 1 monclient: found mon.noname-c 2024-09-13T11:40:18.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 1 -- 172.21.15.5:0/2042638274 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c13ad70 con 0x7fbc0c137660 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc11ed5640 10 monclient: get_auth_request con 0x7fbc0c13b170 auth_method 0 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc11ed5640 1 --2- 172.21.15.5:0/2042638274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c13b170 0x7fbc0c1ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc10ed3640 1 --2- 172.21.15.5:0/2042638274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc0c137660 0x7fbc0c1cdf90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000c4b0 tx=0x7fbc0000c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 1 -- 172.21.15.5:0/2042638274 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00017070 con 0x7fbc0c137660 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 10 monclient: dump: 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.894 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 5 monclient: authenticate success, global_id 4304 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient: _renew_subs 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 1 -- 172.21.15.5:0/2042638274 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc00005d10 con 0x7fbc0c137660 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 -- 172.21.15.5:0/2042638274 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0c131a70 con 0x7fbc0c137660 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 1 -- 172.21.15.5:0/2042638274 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc000072b0 con 0x7fbc0c137660 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc09ffb640 10 monclient: dump: 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient: _renew_subs 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc13938640 1 -- 172.21.15.5:0/2042638274 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc0c131790 con 0x7fbc0c137660 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.879+0000 7fbc0affd640 4 set_mon_vals no callback set 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.883+0000 7fbc13938640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.883+0000 7fbc13938640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.883+0000 7fbc13938640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.883+0000 7fbc13938640 1 -- 172.21.15.5:0/2042638274 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc0c1311d0 con 0x7fbc0c137660 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.883+0000 7fbc09ffb640 1 -- 172.21.15.5:0/2042638274 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fbc0001a070 con 0x7fbc0c137660 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.883+0000 7fbc09ffb640 1 --2- 172.21.15.5:0/2042638274 >> v2:172.21.15.39:6812/1340838781 conn(0x7fbbf803b2a0 0x7fbbf803d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.883+0000 7fbc09ffb640 1 -- 172.21.15.5:0/2042638274 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7fbc0004df80 con 0x7fbc0c137660 2024-09-13T11:40:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.883+0000 7fbc116d4640 1 --2- 172.21.15.5:0/2042638274 >> v2:172.21.15.39:6812/1340838781 conn(0x7fbbf803b2a0 0x7fbbf803d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:18.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.883+0000 7fbc116d4640 10 monclient: get_auth_request con 0x7fbbf803b2a0 auth_method 0 2024-09-13T11:40:18.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.883+0000 7fbc116d4640 1 --2- 172.21.15.5:0/2042638274 >> v2:172.21.15.39:6812/1340838781 conn(0x7fbbf803b2a0 0x7fbbf803d760 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbc04007fd0 tx=0x7fbc04007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:18.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.887+0000 7fbc09ffb640 1 -- 172.21.15.5:0/2042638274 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fbc0c13ad70 con 0x7fbc0c137660 2024-09-13T11:40:18.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.887+0000 7fbc09ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:18.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:18.887+0000 7fbc09ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:19.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.087+0000 7fbc13938640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-13T11:40:19.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.087+0000 7fbc13938640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-13T11:40:19.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.087+0000 7fbc13938640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:19.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.087+0000 7fbc13938640 1 -- 172.21.15.5:0/2042638274 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fbc0c0a1c80 con 0x7fbc0c137660 2024-09-13T11:40:19.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc09ffb640 1 -- 172.21.15.5:0/2042638274 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fbc00011de0 con 0x7fbc0c137660 2024-09-13T11:40:19.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc09ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-13T11:40:19.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc09ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:19.092 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:40:19.092 INFO:teuthology.orchestra.run.smithi005.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T11:40:19.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 1 -- 172.21.15.5:0/2042638274 >> v2:172.21.15.39:6812/1340838781 conn(0x7fbbf803b2a0 msgr2=0x7fbbf803d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 1 --2- 172.21.15.5:0/2042638274 >> v2:172.21.15.39:6812/1340838781 conn(0x7fbbf803b2a0 0x7fbbf803d760 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbc04007fd0 tx=0x7fbc04007e40 comp rx=0 tx=0).stop 2024-09-13T11:40:19.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 10 monclient: shutdown 2024-09-13T11:40:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 1 -- 172.21.15.5:0/2042638274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc0c137660 msgr2=0x7fbc0c1cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 1 --2- 172.21.15.5:0/2042638274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc0c137660 0x7fbc0c1cdf90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000c4b0 tx=0x7fbc0000c970 comp rx=0 tx=0).stop 2024-09-13T11:40:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 1 -- 172.21.15.5:0/2042638274 shutdown_connections 2024-09-13T11:40:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 1 --2- 172.21.15.5:0/2042638274 >> v2:172.21.15.39:6812/1340838781 conn(0x7fbbf803b2a0 0x7fbbf803d760 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 1 --2- 172.21.15.5:0/2042638274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc0c13b170 0x7fbc0c1ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 1 --2- 172.21.15.5:0/2042638274 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc0c137660 0x7fbc0c1cdf90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 1 --2- 172.21.15.5:0/2042638274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc0c134810 0x7fbc0c1cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 1 -- 172.21.15.5:0/2042638274 >> 172.21.15.5:0/2042638274 conn(0x7fbc0c0a2930 msgr2=0x7fbc0c0a3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 1 -- 172.21.15.5:0/2042638274 shutdown_connections 2024-09-13T11:40:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.091+0000 7fbc13938640 1 -- 172.21.15.5:0/2042638274 wait complete. 2024-09-13T11:40:19.109 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T11:40:19.109 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-13T11:40:19.145 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-13T11:40:19.145 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:40:19.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.251+0000 7f7c074d8640 10 monclient: get_monmap_and_config 2024-09-13T11:40:19.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.251+0000 7f7c074d8640 10 monclient: build_initial_monmap 2024-09-13T11:40:19.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.251+0000 7f7c074d8640 10 monclient: monmap: 2024-09-13T11:40:19.255 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:19.255 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.255 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:19.256654+0000 2024-09-13T11:40:19.255 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:19.256654+0000 2024-09-13T11:40:19.255 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:19.255 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.255 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:19.255 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:19.255 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:19.255 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:19.255 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:19.255 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 1 Processor -- start 2024-09-13T11:40:19.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 1 -- start start 2024-09-13T11:40:19.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 10 monclient: init 2024-09-13T11:40:19.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:19.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-13T11:40:19.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c000a0ea0 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:19.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7c000a1ae0 0x7f7c0013a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 10 monclient(hunting): picked mon.noname-a con 0x7f7c000a1ae0 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:19.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7c0013ad30 0x7f7c0013d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 10 monclient(hunting): picked mon.noname-c con 0x7f7c0013ad30 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:19.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c0006f010 con 0x7f7c000a0ea0 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f7c0006f640 con 0x7f7c000a1ae0 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f7c000a0000 con 0x7f7c0013ad30 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c074d8640 10 monclient(hunting): authenticate will time out at 1022.605334s 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7c000a1ae0 0x7f7c0013a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7c000a1ae0 0x7f7c0013a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42364/0 (socket says 172.21.15.5:42364) 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 1 -- 172.21.15.5:0/2540821576 learned_addr learned my addr 172.21.15.5:0/2540821576 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 10 monclient(hunting): get_auth_request con 0x7f7c000a1ae0 auth_method 0 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c05a75640 1 --2- 172.21.15.5:0/2540821576 >> v2:172.21.15.39:3300/0 conn(0x7f7c0013ad30 0x7f7c0013d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c05a75640 10 monclient(hunting): get_auth_request con 0x7f7c0013ad30 auth_method 0 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c05a75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c05a75640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c05a75640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 10 monclient(hunting): handle_auth_done global_id 4401 payload 274 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 1 -- 172.21.15.5:0/2540821576 >> v2:172.21.15.39:3300/0 conn(0x7f7c0013ad30 msgr2=0x7f7c0013d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 1 --2- 172.21.15.5:0/2540821576 >> v2:172.21.15.39:3300/0 conn(0x7f7c0013ad30 0x7f7c0013d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 1 -- 172.21.15.5:0/2540821576 >> v1:172.21.15.39:6789/0 conn(0x7f7c000a0ea0 legacy=0x7f7c000a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 10 monclient: _finish_hunting 0 2024-09-13T11:40:19.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 1 monclient: found mon.noname-a 2024-09-13T11:40:19.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:19.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:19.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.255+0000 7f7c04a73640 1 -- 172.21.15.5:0/2540821576 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0009fd80 con 0x7f7c000a1ae0 2024-09-13T11:40:19.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c04a73640 1 --2- 172.21.15.5:0/2540821576 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c000a1ae0 0x7f7c0013a7f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7c0006fa00 tx=0x7f7bf001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f9459ba8adadc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:19.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 1 -- 172.21.15.5:0/2540821576 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00ab620 con 0x7f7c000a1ae0 2024-09-13T11:40:19.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:19.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 10 monclient: dump: 2024-09-13T11:40:19.267 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.267 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.267 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.267 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.267 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 10 monclient: _finish_auth 0 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 1 -- 172.21.15.5:0/2540821576 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bf00b0070 con 0x7f7c000a1ae0 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 1 -- 172.21.15.5:0/2540821576 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00ab9e0 con 0x7f7c000a1ae0 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 5 monclient: authenticate success, global_id 4401 2024-09-13T11:40:19.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 4 set_mon_vals no callback set 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7bf7fff640 10 monclient: dump: 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.269 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient: shutdown 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 -- 172.21.15.5:0/2540821576 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c000a1ae0 msgr2=0x7f7c0013a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 --2- 172.21.15.5:0/2540821576 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c000a1ae0 0x7f7c0013a7f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7c0006fa00 tx=0x7f7bf001c440 comp rx=0 tx=0).stop 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 -- 172.21.15.5:0/2540821576 shutdown_connections 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 --2- 172.21.15.5:0/2540821576 >> v2:172.21.15.39:3300/0 conn(0x7f7c0013ad30 0x7f7c0013d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 --2- 172.21.15.5:0/2540821576 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c000a1ae0 0x7f7c0013a7f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 -- 172.21.15.5:0/2540821576 >> 172.21.15.5:0/2540821576 conn(0x7f7c000a2930 msgr2=0x7f7c000a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 -- 172.21.15.5:0/2540821576 shutdown_connections 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 -- 172.21.15.5:0/2540821576 wait complete. 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient: build_initial_monmap 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient: monmap: 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:19.270 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 Processor -- start 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 -- start start 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient: init 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:40:19.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c000a0ea0 0x7f7c001cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient(hunting): picked mon.noname-a con 0x7f7c000a0ea0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7c000a1ae0 0x7f7c001324a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient(hunting): picked mon.noname-c con 0x7f7c000a1ae0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c0013ad30 0x7f7c00070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient(hunting): picked mon.noname-b con 0x7f7c0013ad30 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7c00140000 con 0x7f7c000a0ea0 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0013fe80 con 0x7f7c0013ad30 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f7c00140180 con 0x7f7c000a1ae0 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c074d8640 10 monclient(hunting): authenticate will time out at 1022.611565s 2024-09-13T11:40:19.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c04a73640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7c000a1ae0 0x7f7c001324a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c04a73640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7c000a1ae0 0x7f7c001324a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46982/0 (socket says 172.21.15.5:46982) 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05274640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c000a0ea0 0x7f7c001cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c04a73640 1 -- 172.21.15.5:0/610604443 learned_addr learned my addr 172.21.15.5:0/610604443 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c04a73640 10 monclient(hunting): get_auth_request con 0x7f7c000a1ae0 auth_method 0 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c04a73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c04a73640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c04a73640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05274640 10 monclient(hunting): get_auth_request con 0x7f7c000a0ea0 auth_method 0 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05274640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05274640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05274640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c04a73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:19.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c04a73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c04a73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05274640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05274640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05274640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05a75640 1 --2- 172.21.15.5:0/610604443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c0013ad30 0x7f7c00070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05a75640 10 monclient(hunting): get_auth_request con 0x7f7c0013ad30 auth_method 0 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05a75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05a75640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.259+0000 7f7c05a75640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c04a73640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c04a73640 1 -- 172.21.15.5:0/610604443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c0013ad30 msgr2=0x7f7c00070a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c04a73640 1 --2- 172.21.15.5:0/610604443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c0013ad30 0x7f7c00070a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c04a73640 1 -- 172.21.15.5:0/610604443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c000a0ea0 msgr2=0x7f7c001cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c04a73640 1 --2- 172.21.15.5:0/610604443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c000a0ea0 0x7f7c001cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c04a73640 10 monclient: _finish_hunting 0 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c04a73640 1 monclient: found mon.noname-c 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c04a73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c04a73640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c04a73640 1 -- 172.21.15.5:0/610604443 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0011c590 con 0x7f7c000a1ae0 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c04a73640 1 --2- 172.21.15.5:0/610604443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7c000a1ae0 0x7f7c001324a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00ad040 tx=0x7f7bf001ac80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 1 -- 172.21.15.5:0/610604443 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00abd90 con 0x7f7c000a1ae0 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 10 monclient: dump: 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 1 -- 172.21.15.5:0/610604443 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bf00be3f0 con 0x7f7c000a1ae0 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 1 -- 172.21.15.5:0/610604443 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00be7c0 con 0x7f7c000a1ae0 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c074d8640 5 monclient: authenticate success, global_id 4307 2024-09-13T11:40:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c074d8640 10 monclient: _renew_subs 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c074d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c074d8640 1 -- 172.21.15.5:0/610604443 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0011c870 con 0x7f7c000a1ae0 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf77fe640 4 set_mon_vals no callback set 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 10 monclient: dump: 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.277 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c074d8640 10 monclient: _renew_subs 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c074d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c074d8640 1 -- 172.21.15.5:0/610604443 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c0013aa40 con 0x7f7c000a1ae0 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 1 -- 172.21.15.5:0/610604443 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f7bf00c7440 con 0x7f7c000a1ae0 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c074d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c074d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c074d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c074d8640 1 -- 172.21.15.5:0/610604443 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c000a1280 con 0x7f7c000a1ae0 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 1 --2- 172.21.15.5:0/610604443 >> v2:172.21.15.39:6812/1340838781 conn(0x7f7be403b2a0 0x7f7be403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7bf5ffb640 1 -- 172.21.15.5:0/610604443 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f7bf00e9370 con 0x7f7c000a1ae0 2024-09-13T11:40:19.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c05274640 1 --2- 172.21.15.5:0/610604443 >> v2:172.21.15.39:6812/1340838781 conn(0x7f7be403b2a0 0x7f7be403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c05274640 10 monclient: get_auth_request con 0x7f7be403b2a0 auth_method 0 2024-09-13T11:40:19.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.263+0000 7f7c05274640 1 --2- 172.21.15.5:0/610604443 >> v2:172.21.15.39:6812/1340838781 conn(0x7f7be403b2a0 0x7f7be403d760 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc007fd0 tx=0x7f7bfc007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:19.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.267+0000 7f7bf5ffb640 1 -- 172.21.15.5:0/610604443 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f7bf00160f0 con 0x7f7c000a1ae0 2024-09-13T11:40:19.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.267+0000 7f7bf5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:19.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.267+0000 7f7bf5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:19.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.435+0000 7f7c074d8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:19.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.435+0000 7f7c074d8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:19.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.435+0000 7f7c074d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.435+0000 7f7c074d8640 1 -- 172.21.15.5:0/610604443 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c0011c2b0 con 0x7f7c000a1ae0 2024-09-13T11:40:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7bf5ffb640 1 -- 172.21.15.5:0/610604443 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+10870 (secure 0 0 0) 0x7f7bf00b43f0 con 0x7f7c000a1ae0 2024-09-13T11:40:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7bf5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:40:19.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7bf5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:19.439 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:40:19.439 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":20,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","created":"2024-09-13T11:39:53.326128+0000","modified":"2024-09-13T11:40:18.038351+0000","last_up_change":"2024-09-13T11:40:03.374234+0000","last_in_change":"2024-09-13T11:39:55.104089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:40:04.344231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:40:10.361569+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0803d65c-46ec-49b5-a45e-46bcf59ae3f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":4278099835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":4278099835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":4278099835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":4278099835}]},"public_addr":"172.21.15.5:6804/4278099835","cluster_addr":"172.21.15.5:6805/4278099835","heartbeat_back_addr":"172.21.15.5:6807/4278099835","heartbeat_front_addr":"172.21.15.5:6806/4278099835","state":["exists","up"]},{"osd":1,"uuid":"fd592a78-172c-4e1c-9f3e-c49e3e3304d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":1447734001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":1447734001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":1447734001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":1447734001}]},"public_addr":"172.21.15.5:6808/1447734001","cluster_addr":"172.21.15.5:6809/1447734001","heartbeat_back_addr":"172.21.15.5:6811/1447734001","heartbeat_front_addr":"172.21.15.5:6810/1447734001","state":["exists","up"]},{"osd":2,"uuid":"c3cec868-fd4e-472c-83b8-bc2e466671ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":1907498696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":1907498696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":1907498696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":1907498696}]},"public_addr":"172.21.15.5:6800/1907498696","cluster_addr":"172.21.15.5:6801/1907498696","heartbeat_back_addr":"172.21.15.5:6803/1907498696","heartbeat_front_addr":"172.21.15.5:6802/1907498696","state":["exists","up"]},{"osd":3,"uuid":"748f14f6-a1d6-4669-9f81-6b1cd7cd5a60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3188120659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":3188120659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":3188120659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":3188120659}]},"public_addr":"172.21.15.39:6804/3188120659","cluster_addr":"172.21.15.39:6805/3188120659","heartbeat_back_addr":"172.21.15.39:6807/3188120659","heartbeat_front_addr":"172.21.15.39:6806/3188120659","state":["exists","up"]},{"osd":4,"uuid":"f3a05376-7e83-46f9-b3e6-97b31a0942a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":4264535364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":4264535364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":4264535364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4264535364}]},"public_addr":"172.21.15.39:6800/4264535364","cluster_addr":"172.21.15.39:6801/4264535364","heartbeat_back_addr":"172.21.15.39:6803/4264535364","heartbeat_front_addr":"172.21.15.39:6802/4264535364","state":["exists","up"]},{"osd":5,"uuid":"e3b64b2d-c3e5-45d1-90a9-225892061800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1185247880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":1185247880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":1185247880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1185247880}]},"public_addr":"172.21.15.39:6808/1185247880","cluster_addr":"172.21.15.39:6809/1185247880","heartbeat_back_addr":"172.21.15.39:6811/1185247880","heartbeat_front_addr":"172.21.15.39:6810/1185247880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.244929+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.890770+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.206643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.846803+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.612582+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:02.321712+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:40:19.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 1 -- 172.21.15.5:0/610604443 >> v2:172.21.15.39:6812/1340838781 conn(0x7f7be403b2a0 msgr2=0x7f7be403d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 1 --2- 172.21.15.5:0/610604443 >> v2:172.21.15.39:6812/1340838781 conn(0x7f7be403b2a0 0x7f7be403d760 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc007fd0 tx=0x7f7bfc007e40 comp rx=0 tx=0).stop 2024-09-13T11:40:19.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 10 monclient: shutdown 2024-09-13T11:40:19.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:19.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 1 -- 172.21.15.5:0/610604443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7c000a1ae0 msgr2=0x7f7c001324a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 1 --2- 172.21.15.5:0/610604443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7c000a1ae0 0x7f7c001324a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00ad040 tx=0x7f7bf001ac80 comp rx=0 tx=0).stop 2024-09-13T11:40:19.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 1 -- 172.21.15.5:0/610604443 shutdown_connections 2024-09-13T11:40:19.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 1 --2- 172.21.15.5:0/610604443 >> v2:172.21.15.39:6812/1340838781 conn(0x7f7be403b2a0 0x7f7be403d760 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 1 --2- 172.21.15.5:0/610604443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c0013ad30 0x7f7c00070a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 1 --2- 172.21.15.5:0/610604443 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7c000a1ae0 0x7f7c001324a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 1 --2- 172.21.15.5:0/610604443 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c000a0ea0 0x7f7c001cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 1 -- 172.21.15.5:0/610604443 >> 172.21.15.5:0/610604443 conn(0x7f7c000a2930 msgr2=0x7f7c00135640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:19.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 1 -- 172.21.15.5:0/610604443 shutdown_connections 2024-09-13T11:40:19.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.439+0000 7f7c074d8640 1 -- 172.21.15.5:0/610604443 wait complete. 2024-09-13T11:40:19.457 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T11:40:04.344231+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T11:40:10.361569+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T11:40:19.458 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T11:40:19.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.571+0000 7f8bd0ec3640 10 monclient: get_monmap_and_config 2024-09-13T11:40:19.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.571+0000 7f8bd0ec3640 10 monclient: build_initial_monmap 2024-09-13T11:40:19.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.571+0000 7f8bd0ec3640 10 monclient: monmap: 2024-09-13T11:40:19.574 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:19.574 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.574 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:19.576014+0000 2024-09-13T11:40:19.574 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:19.576014+0000 2024-09-13T11:40:19.574 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:19.575 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.575 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:19.575 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:19.575 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:19.575 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:19.575 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:19.575 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 1 Processor -- start 2024-09-13T11:40:19.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 1 -- start start 2024-09-13T11:40:19.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 10 monclient: init 2024-09-13T11:40:19.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8bcc0a1920 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8bcc0a00f0 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8bcc13e270 0x7f8bcc140660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 10 monclient(hunting): picked mon.noname-c con 0x7f8bcc13e270 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bcc0a2410 con 0x7f8bcc0a00f0 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bcc06f670 con 0x7f8bcc0a1920 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f8bcc0a0560 con 0x7f8bcc13e270 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 10 monclient(hunting): authenticate will time out at 1022.925300s 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f8bcc0a00f0 0x7f8bcc13cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:56376/0 (socket says 172.21.15.5:56376) 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 1 -- 172.21.15.5:0/2057354486 learned_addr learned my addr 172.21.15.5:0/2057354486 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:19.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 1 --2- 172.21.15.5:0/2057354486 >> v2:172.21.15.39:3300/0 conn(0x7f8bcc13e270 0x7f8bcc140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): get_auth_request con 0x7f8bcc13e270 auth_method 0 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2413447425 0 0) 0x7f8bcc0a2410 con 0x7f8bcc0a00f0 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient(hunting): _negotiate 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient(hunting): my global_id is 4407 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bac003530 con 0x7f8bcc0a00f0 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:19.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3331735560 0 0) 0x7f8bac003530 con 0x7f8bcc0a00f0 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 >> v2:172.21.15.39:3300/0 conn(0x7f8bcc13e270 msgr2=0x7f8bcc140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 --2- 172.21.15.5:0/2057354486 >> v2:172.21.15.39:3300/0 conn(0x7f8bcc13e270 0x7f8bcc140660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 >> v1:172.21.15.39:6789/0 conn(0x7f8bcc0a1920 legacy=0x7f8bcc0a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient: _finish_hunting 0 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bcc070710 con 0x7f8bcc0a00f0 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient: _finish_auth 0 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient: _check_auth_tickets 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3199362888 0 0) 0x7f8bb4001700 con 0x7f8bcc0a00f0 2024-09-13T11:40:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient: dump: 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:19.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 >> v1:172.21.15.5:6789/0 conn(0x7f8bcc0a00f0 legacy=0x7f8bcc13cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:40:19.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 --2- 172.21.15.5:0/2057354486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bac003a60 0x7f8bac003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient(hunting): picked mon.b con 0x7f8bac003a60 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:19.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 --2- 172.21.15.5:0/2057354486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 0x7f8bac004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient(hunting): picked mon.a con 0x7f8bac004810 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:19.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 --2- 172.21.15.5:0/2057354486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8bac0053d0 0x7f8bac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient(hunting): picked mon.c con 0x7f8bac0053d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:19.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8bcc0a0560 con 0x7f8bac004810 2024-09-13T11:40:19.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac00c240 con 0x7f8bac003a60 2024-09-13T11:40:19.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f8bac00c440 con 0x7f8bac0053d0 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc9d9b640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 1 --2- 172.21.15.5:0/2057354486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 0x7f8bac004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 10 monclient(hunting): get_auth_request con 0x7f8bac004810 auth_method 0 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 1 --2- 172.21.15.5:0/2057354486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8bac0053d0 0x7f8bac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): get_auth_request con 0x7f8bac0053d0 auth_method 0 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bca59c640 1 --2- 172.21.15.5:0/2057354486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bac003a60 0x7f8bac003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bca59c640 10 monclient(hunting): get_auth_request con 0x7f8bac003a60 auth_method 0 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bca59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bca59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 10 monclient(hunting): handle_auth_done global_id 4407 payload 293 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 1 -- 172.21.15.5:0/2057354486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8bac0053d0 msgr2=0x7f8bac00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 1 --2- 172.21.15.5:0/2057354486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8bac0053d0 0x7f8bac00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 1 -- 172.21.15.5:0/2057354486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bac003a60 msgr2=0x7f8bac003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 1 --2- 172.21.15.5:0/2057354486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bac003a60 0x7f8bac003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 10 monclient: _finish_hunting 0 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 1 monclient: found mon.a 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 1 -- 172.21.15.5:0/2057354486 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bac003530 con 0x7f8bac004810 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 10 monclient: _finish_auth 0 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 10 monclient: _check_auth_tickets 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bc1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bcad9d640 1 -- 172.21.15.5:0/2057354486 reap_dead start 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 5 monclient: authenticate success, global_id 4407 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.575+0000 7f8bd0ec3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc1d9b640 1 --2- 172.21.15.5:0/2057354486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 0x7f8bac004c90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4001060 tx=0x7f8bb401cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79fe58866b10318 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4007d20 con 0x7f8bac004810 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc9d9b640 10 monclient: dump: 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bb40a9020 con 0x7f8bac004810 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc9d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc9d9b640 1 -- 172.21.15.5:0/2057354486 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4005b20 con 0x7f8bac004810 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 4 set_mon_vals no callback set 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc9d9b640 10 monclient: dump: 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.590 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.590 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient: shutdown 2024-09-13T11:40:19.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:19.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/2057354486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 msgr2=0x7f8bac004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 --2- 172.21.15.5:0/2057354486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 0x7f8bac004c90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4001060 tx=0x7f8bb401cb40 comp rx=0 tx=0).stop 2024-09-13T11:40:19.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/2057354486 shutdown_connections 2024-09-13T11:40:19.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 --2- 172.21.15.5:0/2057354486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 0x7f8bac004c90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/2057354486 >> 172.21.15.5:0/2057354486 conn(0x7f8bcc0a8720 msgr2=0x7f8bcc0a4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:19.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/2057354486 shutdown_connections 2024-09-13T11:40:19.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/2057354486 wait complete. 2024-09-13T11:40:19.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient: build_initial_monmap 2024-09-13T11:40:19.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient: monmap: 2024-09-13T11:40:19.596 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:19.596 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:19.596 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:19.596 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:19.596 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:19.596 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.596 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:19.596 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 Processor -- start 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 -- start start 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient: init 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 0x7f8bcc13cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient(hunting): picked mon.noname-a con 0x7f8bac004810 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8bcc124230 0x7f8bcc138070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient(hunting): picked mon.noname-c con 0x7f8bcc124230 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bcc1385b0 0x7f8bcc1cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient(hunting): picked mon.noname-b con 0x7f8bcc1385b0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8bcc0a56d0 con 0x7f8bac004810 2024-09-13T11:40:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8bcc0a5550 con 0x7f8bcc1385b0 2024-09-13T11:40:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f8bcc0a4190 con 0x7f8bcc124230 2024-09-13T11:40:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient(hunting): authenticate will time out at 1022.930895s 2024-09-13T11:40:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 0x7f8bcc13cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc1d9b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8bcc124230 0x7f8bcc138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 0x7f8bcc13cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42384/0 (socket says 172.21.15.5:42384) 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 1 -- 172.21.15.5:0/1475643296 learned_addr learned my addr 172.21.15.5:0/1475643296 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 10 monclient(hunting): get_auth_request con 0x7f8bac004810 auth_method 0 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc1d9b640 10 monclient(hunting): get_auth_request con 0x7f8bcc124230 auth_method 0 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bcad9d640 1 --2- 172.21.15.5:0/1475643296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bcc1385b0 0x7f8bcc1cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bcad9d640 10 monclient(hunting): get_auth_request con 0x7f8bcc1385b0 auth_method 0 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bcad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bcad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bcad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 10 monclient(hunting): handle_auth_done global_id 4410 payload 824 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 1 -- 172.21.15.5:0/1475643296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8bcc124230 msgr2=0x7f8bcc138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 1 --2- 172.21.15.5:0/1475643296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8bcc124230 0x7f8bcc138070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 1 -- 172.21.15.5:0/1475643296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bcc1385b0 msgr2=0x7f8bcc1cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 1 --2- 172.21.15.5:0/1475643296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bcc1385b0 0x7f8bcc1cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 10 monclient: _finish_hunting 0 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 1 monclient: found mon.noname-a 2024-09-13T11:40:19.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 1 -- 172.21.15.5:0/1475643296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bcc125470 con 0x7f8bac004810 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc1d9b640 1 --2- 172.21.15.5:0/1475643296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8bcc124230 0x7f8bcc138070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bca59c640 1 --2- 172.21.15.5:0/1475643296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 0x7f8bcc13cd80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00e4e0 tx=0x7f8bbc00e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 1 -- 172.21.15.5:0/1475643296 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc0052e0 con 0x7f8bac004810 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 10 monclient: dump: 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 1 -- 172.21.15.5:0/1475643296 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bbc018070 con 0x7f8bac004810 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 1 -- 172.21.15.5:0/1475643296 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc003840 con 0x7f8bac004810 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 5 monclient: authenticate success, global_id 4410 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient: _renew_subs 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/1475643296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bcc125750 con 0x7f8bac004810 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc959a640 4 set_mon_vals no callback set 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bc37fe640 10 monclient: dump: 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.603 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient: _renew_subs 2024-09-13T11:40:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.579+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/1475643296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bcc09f160 con 0x7f8bac004810 2024-09-13T11:40:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.583+0000 7f8bd0ec3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.583+0000 7f8bd0ec3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.583+0000 7f8bd0ec3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.583+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/1475643296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bcc0a4520 con 0x7f8bac004810 2024-09-13T11:40:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.583+0000 7f8bc37fe640 1 -- 172.21.15.5:0/1475643296 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f8bbc0039e0 con 0x7f8bac004810 2024-09-13T11:40:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.583+0000 7f8bc37fe640 1 --2- 172.21.15.5:0/1475643296 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8bb003b2a0 0x7f8bb003d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.583+0000 7f8bc37fe640 1 -- 172.21.15.5:0/1475643296 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f8bbc04e8b0 con 0x7f8bac004810 2024-09-13T11:40:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.583+0000 7f8bc1d9b640 1 --2- 172.21.15.5:0/1475643296 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8bb003b2a0 0x7f8bb003d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.583+0000 7f8bc1d9b640 10 monclient: get_auth_request con 0x7f8bb003b2a0 auth_method 0 2024-09-13T11:40:19.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.583+0000 7f8bc1d9b640 1 --2- 172.21.15.5:0/1475643296 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8bb003b2a0 0x7f8bb003d760 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8bb40aa000 tx=0x7f8bb4000ff0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:19.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.587+0000 7f8bc37fe640 1 -- 172.21.15.5:0/1475643296 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f8bbc01ddc0 con 0x7f8bac004810 2024-09-13T11:40:19.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.587+0000 7f8bc37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:19.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.587+0000 7f8bc37fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:19.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.747+0000 7f8bd0ec3640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-13T11:40:19.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.751+0000 7f8bd0ec3640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-13T11:40:19.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.751+0000 7f8bd0ec3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:19.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.751+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/1475643296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8bcc06ff80 con 0x7f8bac004810 2024-09-13T11:40:19.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.751+0000 7f8bc37fe640 1 -- 172.21.15.5:0/1475643296 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f8bbc01bd90 con 0x7f8bac004810 2024-09-13T11:40:19.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.751+0000 7f8bc37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-13T11:40:19.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.751+0000 7f8bc37fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:19.752 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 1 2024-09-13T11:40:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/1475643296 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8bb003b2a0 msgr2=0x7f8bb003d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 1 --2- 172.21.15.5:0/1475643296 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8bb003b2a0 0x7f8bb003d760 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8bb40aa000 tx=0x7f8bb4000ff0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 10 monclient: shutdown 2024-09-13T11:40:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/1475643296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 msgr2=0x7f8bcc13cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 1 --2- 172.21.15.5:0/1475643296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 0x7f8bcc13cd80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00e4e0 tx=0x7f8bbc00e9a0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/1475643296 shutdown_connections 2024-09-13T11:40:19.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 1 --2- 172.21.15.5:0/1475643296 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8bb003b2a0 0x7f8bb003d760 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 1 --2- 172.21.15.5:0/1475643296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bcc1385b0 0x7f8bcc1cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 1 --2- 172.21.15.5:0/1475643296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8bcc124230 0x7f8bcc138070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 1 --2- 172.21.15.5:0/1475643296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bac004810 0x7f8bcc13cd80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/1475643296 >> 172.21.15.5:0/1475643296 conn(0x7f8bcc0a8720 msgr2=0x7f8bcc1c3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:19.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/1475643296 shutdown_connections 2024-09-13T11:40:19.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.755+0000 7f8bd0ec3640 1 -- 172.21.15.5:0/1475643296 wait complete. 2024-09-13T11:40:19.771 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-13T11:40:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient: get_monmap_and_config 2024-09-13T11:40:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient: build_initial_monmap 2024-09-13T11:40:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient: monmap: 2024-09-13T11:40:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:19.893029+0000 2024-09-13T11:40:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:19.893029+0000 2024-09-13T11:40:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:19.892 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:19.892 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:19.892 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 1 Processor -- start 2024-09-13T11:40:19.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 1 -- start start 2024-09-13T11:40:19.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient: init 2024-09-13T11:40:19.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:19.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-13T11:40:19.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f897c0a3780 0x7f897c0a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient(hunting): picked mon.noname-a con 0x7f897c0a3780 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f897c0a4ec0 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f897c137fb0 0x7f897c13fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient(hunting): picked mon.noname-b con 0x7f897c137fb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f897c0a5e90 con 0x7f897c0a4ec0 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f897c0a19b0 con 0x7f897c0a3780 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f897c0a1c90 con 0x7f897c137fb0 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8982a36640 10 monclient(hunting): authenticate will time out at 1023.241935s 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897b7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f897c0a4ec0 0x7f897c1378a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:56388/0 (socket says 172.21.15.5:56388) 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897b7fe640 1 -- 172.21.15.5:0/2156531924 learned_addr learned my addr 172.21.15.5:0/2156531924 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 1 --2- 172.21.15.5:0/2156531924 >> v2:172.21.15.5:3300/0 conn(0x7f897c0a3780 0x7f897c0a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8980fd3640 1 --2- 172.21.15.5:0/2156531924 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f897c137fb0 0x7f897c13fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 10 monclient(hunting): get_auth_request con 0x7f897c0a3780 auth_method 0 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8980fd3640 10 monclient(hunting): get_auth_request con 0x7f897c137fb0 auth_method 0 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8980fd3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8980fd3640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8980fd3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897affd640 1 -- 172.21.15.5:0/2156531924 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1097955929 0 0) 0x7f897c0a5e90 con 0x7f897c0a4ec0 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897affd640 10 monclient(hunting): _negotiate 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897affd640 10 monclient(hunting): my global_id is 4416 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897affd640 1 -- 172.21.15.5:0/2156531924 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8964003530 con 0x7f897c0a4ec0 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8980fd3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8980fd3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f8980fd3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 10 monclient(hunting): handle_auth_done global_id 4413 payload 274 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 1 -- 172.21.15.5:0/2156531924 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f897c137fb0 msgr2=0x7f897c13fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 1 --2- 172.21.15.5:0/2156531924 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f897c137fb0 0x7f897c13fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 1 -- 172.21.15.5:0/2156531924 >> v1:172.21.15.5:6789/0 conn(0x7f897c0a4ec0 legacy=0x7f897c1378a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 1 monclient: found mon.noname-a 2024-09-13T11:40:19.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:19.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:19.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.891+0000 7f897bfff640 1 -- 172.21.15.5:0/2156531924 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897c0a4420 con 0x7f897c0a3780 2024-09-13T11:40:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897bfff640 1 --2- 172.21.15.5:0/2156531924 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f897c0a3780 0x7f897c0a3b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f89700097c0 tx=0x7f897001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f29bc571d76165 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 1 -- 172.21.15.5:0/2156531924 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89700b0070 con 0x7f897c0a3780 2024-09-13T11:40:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 10 monclient: dump: 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 10 monclient: _finish_auth 0 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 10 monclient: _check_auth_tickets 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 5 monclient: authenticate success, global_id 4413 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 1 -- 172.21.15.5:0/2156531924 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89700ab600 con 0x7f897c0a3780 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 1 -- 172.21.15.5:0/2156531924 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89700ab9b0 con 0x7f897c0a3780 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 4 set_mon_vals no callback set 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897affd640 10 monclient: dump: 2024-09-13T11:40:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.906 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.906 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.906 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.906 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.906 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.906 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.906 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.906 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.906 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient: shutdown 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 -- 172.21.15.5:0/2156531924 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f897c0a3780 msgr2=0x7f897c0a3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 --2- 172.21.15.5:0/2156531924 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f897c0a3780 0x7f897c0a3b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f89700097c0 tx=0x7f897001c360 comp rx=0 tx=0).stop 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 -- 172.21.15.5:0/2156531924 shutdown_connections 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 --2- 172.21.15.5:0/2156531924 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f897c137fb0 0x7f897c13fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 --2- 172.21.15.5:0/2156531924 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f897c0a3780 0x7f897c0a3b60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 -- 172.21.15.5:0/2156531924 >> 172.21.15.5:0/2156531924 conn(0x7f897c0a6f00 msgr2=0x7f897c0a7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 -- 172.21.15.5:0/2156531924 shutdown_connections 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 -- 172.21.15.5:0/2156531924 wait complete. 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient: build_initial_monmap 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient: monmap: 2024-09-13T11:40:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 Processor -- start 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 -- start start 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient: init 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f897c0a3780 0x7f897c1cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient(hunting): picked mon.noname-b con 0x7f897c0a3780 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f897c137fb0 0x7f897c1d0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient(hunting): picked mon.noname-a con 0x7f897c137fb0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f897c1d0640 0x7f897c13c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient(hunting): picked mon.noname-c con 0x7f897c1d0640 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f897c142ac0 con 0x7f897c137fb0 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f897c142940 con 0x7f897c0a3780 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f897c142c40 con 0x7f897c1d0640 2024-09-13T11:40:19.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8982a36640 10 monclient(hunting): authenticate will time out at 1023.247307s 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f897c1d0640 0x7f897c13c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f897c1d0640 0x7f897c13c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47028/0 (socket says 172.21.15.5:47028) 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 1 -- 172.21.15.5:0/2604806831 learned_addr learned my addr 172.21.15.5:0/2604806831 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 10 monclient(hunting): get_auth_request con 0x7f897c1d0640 auth_method 0 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897b7fe640 1 --2- 172.21.15.5:0/2604806831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f897c137fb0 0x7f897c1d0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897b7fe640 10 monclient(hunting): get_auth_request con 0x7f897c137fb0 auth_method 0 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:19.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897bfff640 1 --2- 172.21.15.5:0/2604806831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f897c0a3780 0x7f897c1cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 1 -- 172.21.15.5:0/2604806831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f897c0a3780 msgr2=0x7f897c1cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 1 --2- 172.21.15.5:0/2604806831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f897c0a3780 0x7f897c1cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 1 -- 172.21.15.5:0/2604806831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f897c137fb0 msgr2=0x7f897c1d0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 1 --2- 172.21.15.5:0/2604806831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f897c137fb0 0x7f897c1d0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 10 monclient: _finish_hunting 0 2024-09-13T11:40:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 1 monclient: found mon.noname-c 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f8980fd3640 1 -- 172.21.15.5:0/2604806831 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897c13f330 con 0x7f897c1d0640 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897bfff640 10 monclient: get_auth_request con 0x7f897c0a3780 auth_method 0 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897b7fe640 1 --2- 172.21.15.5:0/2604806831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f897c137fb0 0x7f897c1d0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.895+0000 7f897bfff640 1 --2- 172.21.15.5:0/2604806831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f897c0a3780 0x7f897c1cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8980fd3640 1 --2- 172.21.15.5:0/2604806831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f897c1d0640 0x7f897c13c4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f897400efc0 tx=0x7f897400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 1 -- 172.21.15.5:0/2604806831 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8974018020 con 0x7f897c1d0640 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 10 monclient: dump: 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 10 monclient: _finish_auth 0 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 1 -- 172.21.15.5:0/2604806831 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8974014480 con 0x7f897c1d0640 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8982a36640 5 monclient: authenticate success, global_id 4316 2024-09-13T11:40:19.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8982a36640 10 monclient: _renew_subs 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8982a36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8982a36640 1 -- 172.21.15.5:0/2604806831 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f897c13fee0 con 0x7f897c1d0640 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 1 -- 172.21.15.5:0/2604806831 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8974014870 con 0x7f897c1d0640 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 10 monclient: dump: 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8982a36640 10 monclient: _renew_subs 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8982a36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8979ffb640 4 set_mon_vals no callback set 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8982a36640 1 -- 172.21.15.5:0/2604806831 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f897c120e40 con 0x7f897c1d0640 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 1 -- 172.21.15.5:0/2604806831 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f897401b020 con 0x7f897c1d0640 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8982a36640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8982a36640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8982a36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8982a36640 1 -- 172.21.15.5:0/2604806831 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f897c0a3b60 con 0x7f897c1d0640 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 1 --2- 172.21.15.5:0/2604806831 >> v2:172.21.15.39:6812/1340838781 conn(0x7f896803b2a0 0x7f896803d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f8978ff9640 1 -- 172.21.15.5:0/2604806831 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f897404fc10 con 0x7f897c1d0640 2024-09-13T11:40:19.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f897bfff640 1 --2- 172.21.15.5:0/2604806831 >> v2:172.21.15.39:6812/1340838781 conn(0x7f896803b2a0 0x7f896803d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:19.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f897bfff640 10 monclient: get_auth_request con 0x7f896803b2a0 auth_method 0 2024-09-13T11:40:19.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.899+0000 7f897bfff640 1 --2- 172.21.15.5:0/2604806831 >> v2:172.21.15.39:6812/1340838781 conn(0x7f896803b2a0 0x7f896803d760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f89700ad000 tx=0x7f89700a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:19.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.903+0000 7f8978ff9640 1 -- 172.21.15.5:0/2604806831 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f8974022030 con 0x7f897c1d0640 2024-09-13T11:40:19.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.903+0000 7f8978ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:19.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:19.903+0000 7f8978ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:20.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.067+0000 7f8982a36640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-13T11:40:20.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.067+0000 7f8982a36640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-13T11:40:20.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.067+0000 7f8982a36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:20.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.067+0000 7f8982a36640 1 -- 172.21.15.5:0/2604806831 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f897c13f960 con 0x7f897c1d0640 2024-09-13T11:40:20.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.067+0000 7f8978ff9640 1 -- 172.21.15.5:0/2604806831 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) ==== 92+0+10 (secure 0 0 0) 0x7f897c13fee0 con 0x7f897c1d0640 2024-09-13T11:40:20.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.067+0000 7f8978ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-13T11:40:20.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.067+0000 7f8978ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:20.070 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 8 2024-09-13T11:40:20.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 1 -- 172.21.15.5:0/2604806831 >> v2:172.21.15.39:6812/1340838781 conn(0x7f896803b2a0 msgr2=0x7f896803d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 1 --2- 172.21.15.5:0/2604806831 >> v2:172.21.15.39:6812/1340838781 conn(0x7f896803b2a0 0x7f896803d760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f89700ad000 tx=0x7f89700a6000 comp rx=0 tx=0).stop 2024-09-13T11:40:20.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 10 monclient: shutdown 2024-09-13T11:40:20.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:20.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 1 -- 172.21.15.5:0/2604806831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f897c1d0640 msgr2=0x7f897c13c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 1 --2- 172.21.15.5:0/2604806831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f897c1d0640 0x7f897c13c4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f897400efc0 tx=0x7f897400c490 comp rx=0 tx=0).stop 2024-09-13T11:40:20.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 1 -- 172.21.15.5:0/2604806831 shutdown_connections 2024-09-13T11:40:20.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 1 --2- 172.21.15.5:0/2604806831 >> v2:172.21.15.39:6812/1340838781 conn(0x7f896803b2a0 0x7f896803d760 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 1 --2- 172.21.15.5:0/2604806831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f897c1d0640 0x7f897c13c4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 1 --2- 172.21.15.5:0/2604806831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f897c137fb0 0x7f897c1d0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 1 --2- 172.21.15.5:0/2604806831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f897c0a3780 0x7f897c1cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 1 -- 172.21.15.5:0/2604806831 >> 172.21.15.5:0/2604806831 conn(0x7f897c0a6f00 msgr2=0x7f897c09bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:20.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 1 -- 172.21.15.5:0/2604806831 shutdown_connections 2024-09-13T11:40:20.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.071+0000 7f8982a36640 1 -- 172.21.15.5:0/2604806831 wait complete. 2024-09-13T11:40:20.088 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-13T11:40:20.089 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:40:20.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.195+0000 7fdb1a1cf640 10 monclient: get_monmap_and_config 2024-09-13T11:40:20.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.195+0000 7fdb1a1cf640 10 monclient: build_initial_monmap 2024-09-13T11:40:20.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.195+0000 7fdb1a1cf640 10 monclient: monmap: 2024-09-13T11:40:20.198 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:20.198 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:20.200060+0000 2024-09-13T11:40:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:20.200060+0000 2024-09-13T11:40:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:20.199 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 1 Processor -- start 2024-09-13T11:40:20.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 1 -- start start 2024-09-13T11:40:20.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 10 monclient: init 2024-09-13T11:40:20.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:20.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-13T11:40:20.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137930 0x7fdb1409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 10 monclient(hunting): picked mon.noname-b con 0x7fdb14137930 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:20.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdb14137d10 0x7fdb141380f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 10 monclient(hunting): picked mon.noname-a con 0x7fdb14137d10 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdb140a4a70 0x7fdb140a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 10 monclient(hunting): picked mon.noname-c con 0x7fdb140a4a70 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fdb140a5fe0 con 0x7fdb14137d10 2024-09-13T11:40:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fdb1409c540 con 0x7fdb140a4a70 2024-09-13T11:40:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fdb1409bef0 con 0x7fdb14137930 2024-09-13T11:40:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 10 monclient(hunting): authenticate will time out at 1023.548851s 2024-09-13T11:40:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb137fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137930 0x7fdb1409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdb14137d10 0x7fdb141380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdb14137d10 0x7fdb141380f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42404/0 (socket says 172.21.15.5:42404) 2024-09-13T11:40:20.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb137fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137930 0x7fdb1409cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47038/0 (socket says 172.21.15.5:47038) 2024-09-13T11:40:20.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 1 -- 172.21.15.5:0/3387051183 learned_addr learned my addr 172.21.15.5:0/3387051183 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:20.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 10 monclient(hunting): get_auth_request con 0x7fdb14137d10 auth_method 0 2024-09-13T11:40:20.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:20.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:20.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:20.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb137fe640 10 monclient(hunting): get_auth_request con 0x7fdb14137930 auth_method 0 2024-09-13T11:40:20.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:20.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb137fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:20.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:20.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:20.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:20.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:20.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:20.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:20.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:20.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb13fff640 1 --2- 172.21.15.5:0/3387051183 >> v2:172.21.15.39:3300/0 conn(0x7fdb140a4a70 0x7fdb140a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 10 monclient(hunting): handle_auth_done global_id 4422 payload 274 2024-09-13T11:40:20.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 1 -- 172.21.15.5:0/3387051183 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137930 msgr2=0x7fdb1409cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:20.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 1 --2- 172.21.15.5:0/3387051183 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137930 0x7fdb1409cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 1 -- 172.21.15.5:0/3387051183 >> v2:172.21.15.39:3300/0 conn(0x7fdb140a4a70 msgr2=0x7fdb140a4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:20.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 1 --2- 172.21.15.5:0/3387051183 >> v2:172.21.15.39:3300/0 conn(0x7fdb140a4a70 0x7fdb140a4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 10 monclient: _finish_hunting 0 2024-09-13T11:40:20.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 1 monclient: found mon.noname-a 2024-09-13T11:40:20.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:20.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:20.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 1 -- 172.21.15.5:0/3387051183 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1409c1d0 con 0x7fdb14137d10 2024-09-13T11:40:20.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb13fff640 10 monclient: get_auth_request con 0x7fdb140a4a70 auth_method 0 2024-09-13T11:40:20.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb13fff640 1 --2- 172.21.15.5:0/3387051183 >> v2:172.21.15.39:3300/0 conn(0x7fdb140a4a70 0x7fdb140a4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:20.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb137fe640 1 --2- 172.21.15.5:0/3387051183 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137930 0x7fdb1409cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:20.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb12ffd640 1 --2- 172.21.15.5:0/3387051183 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb14137d10 0x7fdb141380f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdafc009870 tx=0x7fdafc01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada77e69e81d87f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:20.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 1 -- 172.21.15.5:0/3387051183 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdafc0b0070 con 0x7fdb14137d10 2024-09-13T11:40:20.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:20.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:20.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 10 monclient: dump: 2024-09-13T11:40:20.211 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:20.211 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.211 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.211 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.211 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:20.211 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.211 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:20.212 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:20.212 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:20.212 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:20.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:20.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 1 -- 172.21.15.5:0/3387051183 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdafc0ab600 con 0x7fdb14137d10 2024-09-13T11:40:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 5 monclient: authenticate success, global_id 4422 2024-09-13T11:40:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 1 -- 172.21.15.5:0/3387051183 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdafc0ab9d0 con 0x7fdb14137d10 2024-09-13T11:40:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb127fc640 10 monclient: dump: 2024-09-13T11:40:20.214 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:20.214 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.214 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.214 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.214 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:20.214 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.214 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:20.214 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:20.215 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:20.215 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:20.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.199+0000 7fdb1a1cf640 4 set_mon_vals no callback set 2024-09-13T11:40:20.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient: shutdown 2024-09-13T11:40:20.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:20.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3387051183 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb14137d10 msgr2=0x7fdb141380f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 --2- 172.21.15.5:0/3387051183 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb14137d10 0x7fdb141380f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdafc009870 tx=0x7fdafc01c380 comp rx=0 tx=0).stop 2024-09-13T11:40:20.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3387051183 shutdown_connections 2024-09-13T11:40:20.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 --2- 172.21.15.5:0/3387051183 >> v2:172.21.15.39:3300/0 conn(0x7fdb140a4a70 0x7fdb140a4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 --2- 172.21.15.5:0/3387051183 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb14137d10 0x7fdb141380f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 --2- 172.21.15.5:0/3387051183 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137930 0x7fdb1409cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3387051183 >> 172.21.15.5:0/3387051183 conn(0x7fdb140a6c70 msgr2=0x7fdb140a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:20.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3387051183 shutdown_connections 2024-09-13T11:40:20.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3387051183 wait complete. 2024-09-13T11:40:20.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient: build_initial_monmap 2024-09-13T11:40:20.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient: monmap: 2024-09-13T11:40:20.225 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:20.225 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:20.225 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:20.225 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:20.225 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:20.226 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 Processor -- start 2024-09-13T11:40:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 -- start start 2024-09-13T11:40:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient: init 2024-09-13T11:40:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:40:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb140a4a70 0x7fdb141cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient(hunting): picked mon.noname-a con 0x7fdb140a4a70 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb14137930 0x7fdb141369f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient(hunting): picked mon.noname-b con 0x7fdb14137930 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137d10 0x7fdb1413bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient(hunting): picked mon.noname-c con 0x7fdb14137d10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fdb14141190 con 0x7fdb140a4a70 2024-09-13T11:40:20.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdb14141010 con 0x7fdb14137930 2024-09-13T11:40:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fdb14141310 con 0x7fdb14137d10 2024-09-13T11:40:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient(hunting): authenticate will time out at 1023.554105s 2024-09-13T11:40:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb13fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137d10 0x7fdb1413bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb140a4a70 0x7fdb141cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb13fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137d10 0x7fdb1413bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47044/0 (socket says 172.21.15.5:47044) 2024-09-13T11:40:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb140a4a70 0x7fdb141cf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42418/0 (socket says 172.21.15.5:42418) 2024-09-13T11:40:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb13fff640 1 -- 172.21.15.5:0/3190227485 learned_addr learned my addr 172.21.15.5:0/3190227485 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb13fff640 10 monclient(hunting): get_auth_request con 0x7fdb14137d10 auth_method 0 2024-09-13T11:40:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb13fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 10 monclient(hunting): get_auth_request con 0x7fdb140a4a70 auth_method 0 2024-09-13T11:40:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb12ffd640 1 --2- 172.21.15.5:0/3190227485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb14137930 0x7fdb141369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 10 monclient(hunting): handle_auth_done global_id 4425 payload 824 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 1 -- 172.21.15.5:0/3190227485 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137d10 msgr2=0x7fdb1413bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 1 --2- 172.21.15.5:0/3190227485 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137d10 0x7fdb1413bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 1 -- 172.21.15.5:0/3190227485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb14137930 msgr2=0x7fdb141369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 1 --2- 172.21.15.5:0/3190227485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb14137930 0x7fdb141369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 10 monclient: _finish_hunting 0 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 1 monclient: found mon.noname-a 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 1 -- 172.21.15.5:0/3190227485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb14122390 con 0x7fdb140a4a70 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb12ffd640 10 monclient: get_auth_request con 0x7fdb14137930 auth_method 0 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb13fff640 1 --2- 172.21.15.5:0/3190227485 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137d10 0x7fdb1413bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb12ffd640 1 --2- 172.21.15.5:0/3190227485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb14137930 0x7fdb141369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb137fe640 1 --2- 172.21.15.5:0/3190227485 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb140a4a70 0x7fdb141cf580 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400e9b0 tx=0x7fdb0400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 1 -- 172.21.15.5:0/3190227485 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04017070 con 0x7fdb140a4a70 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 10 monclient: dump: 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:20.232 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 10 monclient: _finish_auth 0 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 1 -- 172.21.15.5:0/3190227485 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb0400c9a0 con 0x7fdb140a4a70 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 1 -- 172.21.15.5:0/3190227485 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0400cd90 con 0x7fdb140a4a70 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb0bfff640 10 monclient: dump: 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb11ffb640 4 set_mon_vals no callback set 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 5 monclient: authenticate success, global_id 4425 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient: _renew_subs 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3190227485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb14120d10 con 0x7fdb140a4a70 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient: _renew_subs 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.203+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3190227485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb140a0d90 con 0x7fdb140a4a70 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.207+0000 7fdb1a1cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.207+0000 7fdb1a1cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.207+0000 7fdb1a1cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.207+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3190227485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb14121c60 con 0x7fdb140a4a70 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.207+0000 7fdb0bfff640 1 -- 172.21.15.5:0/3190227485 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fdb0401a070 con 0x7fdb140a4a70 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.207+0000 7fdb0bfff640 1 --2- 172.21.15.5:0/3190227485 >> v2:172.21.15.39:6812/1340838781 conn(0x7fdaf403b2a0 0x7fdaf403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.207+0000 7fdb0bfff640 1 -- 172.21.15.5:0/3190227485 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7fdb0404eab0 con 0x7fdb140a4a70 2024-09-13T11:40:20.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.207+0000 7fdb12ffd640 1 --2- 172.21.15.5:0/3190227485 >> v2:172.21.15.39:6812/1340838781 conn(0x7fdaf403b2a0 0x7fdaf403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.207+0000 7fdb12ffd640 10 monclient: get_auth_request con 0x7fdaf403b2a0 auth_method 0 2024-09-13T11:40:20.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.207+0000 7fdb12ffd640 1 --2- 172.21.15.5:0/3190227485 >> v2:172.21.15.39:6812/1340838781 conn(0x7fdaf403b2a0 0x7fdaf403d760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0ad000 tx=0x7fdafc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:20.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.211+0000 7fdb0bfff640 1 -- 172.21.15.5:0/3190227485 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fdb14121c60 con 0x7fdb140a4a70 2024-09-13T11:40:20.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.211+0000 7fdb0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:20.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.211+0000 7fdb0bfff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:20.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.371+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3190227485 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdb1409cc90 con 0x7fdaf403b2a0 2024-09-13T11:40:20.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.375+0000 7fdb0bfff640 1 -- 172.21.15.5:0/3190227485 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34047 (secure 0 0 0) 0x7fdb1409cc90 con 0x7fdaf403b2a0 2024-09-13T11:40:20.377 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:40:20.377 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T11:40:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3190227485 >> v2:172.21.15.39:6812/1340838781 conn(0x7fdaf403b2a0 msgr2=0x7fdaf403d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 1 --2- 172.21.15.5:0/3190227485 >> v2:172.21.15.39:6812/1340838781 conn(0x7fdaf403b2a0 0x7fdaf403d760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0ad000 tx=0x7fdafc008040 comp rx=0 tx=0).stop 2024-09-13T11:40:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 10 monclient: shutdown 2024-09-13T11:40:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3190227485 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb140a4a70 msgr2=0x7fdb141cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 1 --2- 172.21.15.5:0/3190227485 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb140a4a70 0x7fdb141cf580 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400e9b0 tx=0x7fdb0400ee70 comp rx=0 tx=0).stop 2024-09-13T11:40:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3190227485 shutdown_connections 2024-09-13T11:40:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 1 --2- 172.21.15.5:0/3190227485 >> v2:172.21.15.39:6812/1340838781 conn(0x7fdaf403b2a0 0x7fdaf403d760 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 1 --2- 172.21.15.5:0/3190227485 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb14137d10 0x7fdb1413bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 1 --2- 172.21.15.5:0/3190227485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb14137930 0x7fdb141369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 1 --2- 172.21.15.5:0/3190227485 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb140a4a70 0x7fdb141cf580 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3190227485 >> 172.21.15.5:0/3190227485 conn(0x7fdb140a6c70 msgr2=0x7fdb140a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:20.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3190227485 shutdown_connections 2024-09-13T11:40:20.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.379+0000 7fdb1a1cf640 1 -- 172.21.15.5:0/3190227485 wait complete. 2024-09-13T11:40:20.394 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-13T11:40:20.300288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167440,"kb_used_data":1096,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063600,"statfs":{"total":579820584960,"available":579649126400,"internally_reserved":0,"allocated":1122304,"data_stored":1377370,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.181402"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T11:40:18.047595+0000","last_change":"2024-09-13T11:40:14.135514+0000","last_active":"2024-09-13T11:40:18.047595+0000","last_peered":"2024-09-13T11:40:18.047595+0000","last_clean":"2024-09-13T11:40:18.047595+0000","last_became_active":"2024-09-13T11:40:12.461041+0000","last_became_peered":"2024-09-13T11:40:12.461041+0000","last_unstale":"2024-09-13T11:40:18.047595+0000","last_undegraded":"2024-09-13T11:40:18.047595+0000","last_fullsized":"2024-09-13T11:40:18.047595+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:50.409094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126878,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T11:40:18.044991+0000","last_change":"2024-09-13T11:40:14.143285+0000","last_active":"2024-09-13T11:40:18.044991+0000","last_peered":"2024-09-13T11:40:18.044991+0000","last_clean":"2024-09-13T11:40:18.044991+0000","last_became_active":"2024-09-13T11:40:12.106972+0000","last_became_peered":"2024-09-13T11:40:12.106972+0000","last_unstale":"2024-09-13T11:40:18.044991+0000","last_undegraded":"2024-09-13T11:40:18.044991+0000","last_fullsized":"2024-09-13T11:40:18.044991+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:29.565401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001297091,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T11:40:18.047521+0000","last_change":"2024-09-13T11:40:14.135429+0000","last_active":"2024-09-13T11:40:18.047521+0000","last_peered":"2024-09-13T11:40:18.047521+0000","last_clean":"2024-09-13T11:40:18.047521+0000","last_became_active":"2024-09-13T11:40:12.120816+0000","last_became_peered":"2024-09-13T11:40:12.120816+0000","last_unstale":"2024-09-13T11:40:18.047521+0000","last_undegraded":"2024-09-13T11:40:18.047521+0000","last_fullsized":"2024-09-13T11:40:18.047521+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:42.012635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011926969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T11:40:18.044982+0000","last_change":"2024-09-13T11:40:14.143208+0000","last_active":"2024-09-13T11:40:18.044982+0000","last_peered":"2024-09-13T11:40:18.044982+0000","last_clean":"2024-09-13T11:40:18.044982+0000","last_became_active":"2024-09-13T11:40:12.120092+0000","last_became_peered":"2024-09-13T11:40:12.120092+0000","last_unstale":"2024-09-13T11:40:18.044982+0000","last_undegraded":"2024-09-13T11:40:18.044982+0000","last_fullsized":"2024-09-13T11:40:18.044982+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:10.230543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001217161,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.140927+0000","last_change":"2024-09-13T11:40:14.139720+0000","last_active":"2024-09-13T11:40:14.140927+0000","last_peered":"2024-09-13T11:40:14.140927+0000","last_clean":"2024-09-13T11:40:14.140927+0000","last_became_active":"2024-09-13T11:40:12.110703+0000","last_became_peered":"2024-09-13T11:40:12.110703+0000","last_unstale":"2024-09-13T11:40:14.140927+0000","last_undegraded":"2024-09-13T11:40:14.140927+0000","last_fullsized":"2024-09-13T11:40:14.140927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:57.779487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021711669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.130605+0000","last_change":"2024-09-13T11:40:14.131038+0000","last_active":"2024-09-13T11:40:14.130605+0000","last_peered":"2024-09-13T11:40:14.130605+0000","last_clean":"2024-09-13T11:40:14.130605+0000","last_became_active":"2024-09-13T11:40:12.102142+0000","last_became_peered":"2024-09-13T11:40:12.102142+0000","last_unstale":"2024-09-13T11:40:14.130605+0000","last_undegraded":"2024-09-13T11:40:14.130605+0000","last_fullsized":"2024-09-13T11:40:14.130605+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:15.679795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046429,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T11:40:18.047468+0000","last_change":"2024-09-13T11:40:14.135976+0000","last_active":"2024-09-13T11:40:18.047468+0000","last_peered":"2024-09-13T11:40:18.047468+0000","last_clean":"2024-09-13T11:40:18.047468+0000","last_became_active":"2024-09-13T11:40:12.105280+0000","last_became_peered":"2024-09-13T11:40:12.105280+0000","last_unstale":"2024-09-13T11:40:18.047468+0000","last_undegraded":"2024-09-13T11:40:18.047468+0000","last_fullsized":"2024-09-13T11:40:18.047468+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:20.791570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001476088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.138397+0000","last_change":"2024-09-13T11:40:14.138397+0000","last_active":"2024-09-13T11:40:14.138397+0000","last_peered":"2024-09-13T11:40:14.138397+0000","last_clean":"2024-09-13T11:40:14.138397+0000","last_became_active":"2024-09-13T11:40:12.113752+0000","last_became_peered":"2024-09-13T11:40:12.113752+0000","last_unstale":"2024-09-13T11:40:14.138397+0000","last_undegraded":"2024-09-13T11:40:14.138397+0000","last_fullsized":"2024-09-13T11:40:14.138397+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:50:52.508777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086960399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":113,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T11:40:18.047440+0000","last_change":"2024-09-13T11:40:05.394873+0000","last_active":"2024-09-13T11:40:18.047440+0000","last_peered":"2024-09-13T11:40:18.047440+0000","last_clean":"2024-09-13T11:40:18.047440+0000","last_became_active":"2024-09-13T11:40:05.394176+0000","last_became_peered":"2024-09-13T11:40:05.394176+0000","last_unstale":"2024-09-13T11:40:18.047440+0000","last_undegraded":"2024-09-13T11:40:18.047440+0000","last_fullsized":"2024-09-13T11:40:18.047440+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_clean_scrub_stamp":"2024-09-13T11:40:04.377013+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:00:31.498589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":584682,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":55773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":580920,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:40:20.395 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:40:20.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 10 monclient: get_monmap_and_config 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 10 monclient: build_initial_monmap 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 10 monclient: monmap: 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:20.510825+0000 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:20.510825+0000 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:20.509 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:20.510 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 1 Processor -- start 2024-09-13T11:40:20.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 1 -- start start 2024-09-13T11:40:20.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 10 monclient: init 2024-09-13T11:40:20.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:20.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-13T11:40:20.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98941309b0 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:20.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9894136e50 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:20.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f989413a960 0x7f989413cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 10 monclient(hunting): picked mon.noname-a con 0x7f989413a960 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:20.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f989409e6b0 con 0x7f9894136e50 2024-09-13T11:40:20.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9894130560 con 0x7f98941309b0 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f98941314a0 con 0x7f989413a960 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9899c4b640 10 monclient(hunting): authenticate will time out at 1023.859642s 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9892ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f9894136e50 0x7f9894139240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:56398/0 (socket says 172.21.15.5:56398) 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9892ffd640 1 -- 172.21.15.5:0/1672003280 learned_addr learned my addr 172.21.15.5:0/1672003280 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9893fff640 1 --2- 172.21.15.5:0/1672003280 >> v2:172.21.15.5:3300/0 conn(0x7f989413a960 0x7f989413cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9893fff640 10 monclient(hunting): get_auth_request con 0x7f989413a960 auth_method 0 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9893fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9893fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9893fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9893fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9893fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:20.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.507+0000 7f9893fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:20.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9893fff640 10 monclient(hunting): handle_auth_done global_id 4428 payload 274 2024-09-13T11:40:20.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9893fff640 1 -- 172.21.15.5:0/1672003280 >> v1:172.21.15.39:6789/0 conn(0x7f98941309b0 legacy=0x7f9894130d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:20.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9893fff640 1 -- 172.21.15.5:0/1672003280 >> v1:172.21.15.5:6789/0 conn(0x7f9894136e50 legacy=0x7f9894139240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9893fff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:20.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9893fff640 1 monclient: found mon.noname-a 2024-09-13T11:40:20.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9893fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:20.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9893fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:20.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9893fff640 1 -- 172.21.15.5:0/1672003280 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f989409e990 con 0x7f989413a960 2024-09-13T11:40:20.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9893fff640 1 --2- 172.21.15.5:0/1672003280 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f989413a960 0x7f989413cd50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f988c009d90 tx=0x7f988c01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2826994cdb9b2218 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:20.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 1 -- 172.21.15.5:0/1672003280 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f988c0b2070 con 0x7f989413a960 2024-09-13T11:40:20.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:20.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:20.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 10 monclient: dump: 2024-09-13T11:40:20.519 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:20.519 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.519 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.519 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.519 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:20.519 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 1 -- 172.21.15.5:0/1672003280 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f988c0a5780 con 0x7f989413a960 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 1 -- 172.21.15.5:0/1672003280 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f988c0a5b50 con 0x7f989413a960 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 5 monclient: authenticate success, global_id 4428 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 4 set_mon_vals no callback set 2024-09-13T11:40:20.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f98927fc640 10 monclient: dump: 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 10 monclient: shutdown 2024-09-13T11:40:20.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 1 -- 172.21.15.5:0/1672003280 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f989413a960 msgr2=0x7f989413cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 1 --2- 172.21.15.5:0/1672003280 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f989413a960 0x7f989413cd50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f988c009d90 tx=0x7f988c01c8c0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 1 -- 172.21.15.5:0/1672003280 shutdown_connections 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 1 --2- 172.21.15.5:0/1672003280 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f989413a960 0x7f989413cd50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 1 -- 172.21.15.5:0/1672003280 >> 172.21.15.5:0/1672003280 conn(0x7f98940a2930 msgr2=0x7f98940a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 1 -- 172.21.15.5:0/1672003280 shutdown_connections 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 1 -- 172.21.15.5:0/1672003280 wait complete. 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 10 monclient: build_initial_monmap 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.511+0000 7f9899c4b640 10 monclient: monmap: 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:20.522 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 1 Processor -- start 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 1 -- start start 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient: init 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98941309b0 0x7f98941cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient(hunting): picked mon.noname-c con 0x7f98941309b0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:20.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9894136e50 0x7f98941d0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient(hunting): picked mon.noname-b con 0x7f9894136e50 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f989413a960 0x7f9894073450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient(hunting): picked mon.noname-a con 0x7f989413a960 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f98940a0660 con 0x7f989413a960 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f989409f870 con 0x7f9894136e50 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f98940a07e0 con 0x7f98941309b0 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient(hunting): authenticate will time out at 1023.864677s 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98941309b0 0x7f98941cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9893fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f989413a960 0x7f9894073450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98941309b0 0x7f98941cfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47054/0 (socket says 172.21.15.5:47054) 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 1 -- 172.21.15.5:0/1206714933 learned_addr learned my addr 172.21.15.5:0/1206714933 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9893fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f989413a960 0x7f9894073450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42436/0 (socket says 172.21.15.5:42436) 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 10 monclient(hunting): get_auth_request con 0x7f98941309b0 auth_method 0 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9893fff640 10 monclient(hunting): get_auth_request con 0x7f989413a960 auth_method 0 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9893fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9893fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9893fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:20.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9893fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9893fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9893fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 10 monclient(hunting): handle_auth_done global_id 4325 payload 824 2024-09-13T11:40:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9892ffd640 1 --2- 172.21.15.5:0/1206714933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9894136e50 0x7f98941d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 1 -- 172.21.15.5:0/1206714933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9894136e50 msgr2=0x7f98941d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 1 --2- 172.21.15.5:0/1206714933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9894136e50 0x7f98941d0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 1 -- 172.21.15.5:0/1206714933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f989413a960 msgr2=0x7f9894073450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 1 --2- 172.21.15.5:0/1206714933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f989413a960 0x7f9894073450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 10 monclient: _finish_hunting 0 2024-09-13T11:40:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 1 monclient: found mon.noname-c 2024-09-13T11:40:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 1 -- 172.21.15.5:0/1206714933 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f989409c290 con 0x7f98941309b0 2024-09-13T11:40:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9893fff640 1 --2- 172.21.15.5:0/1206714933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f989413a960 0x7f9894073450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f98937fe640 1 --2- 172.21.15.5:0/1206714933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98941309b0 0x7f98941cfe10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f987c00c450 tx=0x7f987c00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 1 -- 172.21.15.5:0/1206714933 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c012830 con 0x7f98941309b0 2024-09-13T11:40:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 10 monclient: dump: 2024-09-13T11:40:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.528 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:20.528 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.528 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:20.528 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:20.528 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:20.528 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 10 monclient: _finish_auth 0 2024-09-13T11:40:20.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 1 -- 172.21.15.5:0/1206714933 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f987c017070 con 0x7f98941309b0 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 5 monclient: authenticate success, global_id 4325 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient: _renew_subs 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 1 -- 172.21.15.5:0/1206714933 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f989406dbf0 con 0x7f98941309b0 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 1 -- 172.21.15.5:0/1206714933 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c012c30 con 0x7f98941309b0 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 10 monclient: dump: 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.530 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.530 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.530 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:20.530 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.530 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:20.530 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:20.530 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:20.530 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9891ffb640 4 set_mon_vals no callback set 2024-09-13T11:40:20.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient: _renew_subs 2024-09-13T11:40:20.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:20.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 1 -- 172.21.15.5:0/1206714933 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9894140440 con 0x7f98941309b0 2024-09-13T11:40:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 1 -- 172.21.15.5:0/1206714933 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f987c01c460 con 0x7f98941309b0 2024-09-13T11:40:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9899c4b640 1 -- 172.21.15.5:0/1206714933 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98940a0c20 con 0x7f98941309b0 2024-09-13T11:40:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 1 --2- 172.21.15.5:0/1206714933 >> v2:172.21.15.39:6812/1340838781 conn(0x7f987403b2a0 0x7f987403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f988bfff640 1 -- 172.21.15.5:0/1206714933 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f987c04ea00 con 0x7f98941309b0 2024-09-13T11:40:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9892ffd640 1 --2- 172.21.15.5:0/1206714933 >> v2:172.21.15.39:6812/1340838781 conn(0x7f987403b2a0 0x7f987403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9892ffd640 10 monclient: get_auth_request con 0x7f987403b2a0 auth_method 0 2024-09-13T11:40:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.515+0000 7f9892ffd640 1 --2- 172.21.15.5:0/1206714933 >> v2:172.21.15.39:6812/1340838781 conn(0x7f987403b2a0 0x7f987403d760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f98941366d0 tx=0x7f9884009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:20.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.519+0000 7f988bfff640 1 -- 172.21.15.5:0/1206714933 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f987c012dd0 con 0x7f98941309b0 2024-09-13T11:40:20.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.519+0000 7f988bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:20.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.519+0000 7f988bfff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:20.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.679+0000 7f9899c4b640 1 -- 172.21.15.5:0/1206714933 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9894069ff0 con 0x7f987403b2a0 2024-09-13T11:40:20.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f988bfff640 1 -- 172.21.15.5:0/1206714933 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34047 (secure 0 0 0) 0x7f9894069ff0 con 0x7f987403b2a0 2024-09-13T11:40:20.685 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:40:20.685 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T11:40:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 1 -- 172.21.15.5:0/1206714933 >> v2:172.21.15.39:6812/1340838781 conn(0x7f987403b2a0 msgr2=0x7f987403d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 1 --2- 172.21.15.5:0/1206714933 >> v2:172.21.15.39:6812/1340838781 conn(0x7f987403b2a0 0x7f987403d760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f98941366d0 tx=0x7f9884009040 comp rx=0 tx=0).stop 2024-09-13T11:40:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 10 monclient: shutdown 2024-09-13T11:40:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 1 -- 172.21.15.5:0/1206714933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98941309b0 msgr2=0x7f98941cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 1 --2- 172.21.15.5:0/1206714933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98941309b0 0x7f98941cfe10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f987c00c450 tx=0x7f987c00c910 comp rx=0 tx=0).stop 2024-09-13T11:40:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 1 -- 172.21.15.5:0/1206714933 shutdown_connections 2024-09-13T11:40:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 1 --2- 172.21.15.5:0/1206714933 >> v2:172.21.15.39:6812/1340838781 conn(0x7f987403b2a0 0x7f987403d760 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 1 --2- 172.21.15.5:0/1206714933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f989413a960 0x7f9894073450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 1 --2- 172.21.15.5:0/1206714933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9894136e50 0x7f98941d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 1 --2- 172.21.15.5:0/1206714933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f98941309b0 0x7f98941cfe10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 1 -- 172.21.15.5:0/1206714933 >> 172.21.15.5:0/1206714933 conn(0x7f98940a2930 msgr2=0x7f989412f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:20.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.683+0000 7f9899c4b640 1 -- 172.21.15.5:0/1206714933 shutdown_connections 2024-09-13T11:40:20.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.687+0000 7f9899c4b640 1 -- 172.21.15.5:0/1206714933 wait complete. 2024-09-13T11:40:20.706 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-13T11:40:20.300288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167440,"kb_used_data":1096,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063600,"statfs":{"total":579820584960,"available":579649126400,"internally_reserved":0,"allocated":1122304,"data_stored":1377370,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.181402"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T11:40:18.047595+0000","last_change":"2024-09-13T11:40:14.135514+0000","last_active":"2024-09-13T11:40:18.047595+0000","last_peered":"2024-09-13T11:40:18.047595+0000","last_clean":"2024-09-13T11:40:18.047595+0000","last_became_active":"2024-09-13T11:40:12.461041+0000","last_became_peered":"2024-09-13T11:40:12.461041+0000","last_unstale":"2024-09-13T11:40:18.047595+0000","last_undegraded":"2024-09-13T11:40:18.047595+0000","last_fullsized":"2024-09-13T11:40:18.047595+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:50.409094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126878,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T11:40:18.044991+0000","last_change":"2024-09-13T11:40:14.143285+0000","last_active":"2024-09-13T11:40:18.044991+0000","last_peered":"2024-09-13T11:40:18.044991+0000","last_clean":"2024-09-13T11:40:18.044991+0000","last_became_active":"2024-09-13T11:40:12.106972+0000","last_became_peered":"2024-09-13T11:40:12.106972+0000","last_unstale":"2024-09-13T11:40:18.044991+0000","last_undegraded":"2024-09-13T11:40:18.044991+0000","last_fullsized":"2024-09-13T11:40:18.044991+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:29.565401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001297091,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T11:40:18.047521+0000","last_change":"2024-09-13T11:40:14.135429+0000","last_active":"2024-09-13T11:40:18.047521+0000","last_peered":"2024-09-13T11:40:18.047521+0000","last_clean":"2024-09-13T11:40:18.047521+0000","last_became_active":"2024-09-13T11:40:12.120816+0000","last_became_peered":"2024-09-13T11:40:12.120816+0000","last_unstale":"2024-09-13T11:40:18.047521+0000","last_undegraded":"2024-09-13T11:40:18.047521+0000","last_fullsized":"2024-09-13T11:40:18.047521+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:42.012635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011926969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T11:40:18.044982+0000","last_change":"2024-09-13T11:40:14.143208+0000","last_active":"2024-09-13T11:40:18.044982+0000","last_peered":"2024-09-13T11:40:18.044982+0000","last_clean":"2024-09-13T11:40:18.044982+0000","last_became_active":"2024-09-13T11:40:12.120092+0000","last_became_peered":"2024-09-13T11:40:12.120092+0000","last_unstale":"2024-09-13T11:40:18.044982+0000","last_undegraded":"2024-09-13T11:40:18.044982+0000","last_fullsized":"2024-09-13T11:40:18.044982+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:10.230543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001217161,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.140927+0000","last_change":"2024-09-13T11:40:14.139720+0000","last_active":"2024-09-13T11:40:14.140927+0000","last_peered":"2024-09-13T11:40:14.140927+0000","last_clean":"2024-09-13T11:40:14.140927+0000","last_became_active":"2024-09-13T11:40:12.110703+0000","last_became_peered":"2024-09-13T11:40:12.110703+0000","last_unstale":"2024-09-13T11:40:14.140927+0000","last_undegraded":"2024-09-13T11:40:14.140927+0000","last_fullsized":"2024-09-13T11:40:14.140927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:57.779487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021711669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.130605+0000","last_change":"2024-09-13T11:40:14.131038+0000","last_active":"2024-09-13T11:40:14.130605+0000","last_peered":"2024-09-13T11:40:14.130605+0000","last_clean":"2024-09-13T11:40:14.130605+0000","last_became_active":"2024-09-13T11:40:12.102142+0000","last_became_peered":"2024-09-13T11:40:12.102142+0000","last_unstale":"2024-09-13T11:40:14.130605+0000","last_undegraded":"2024-09-13T11:40:14.130605+0000","last_fullsized":"2024-09-13T11:40:14.130605+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:15.679795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046429,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T11:40:18.047468+0000","last_change":"2024-09-13T11:40:14.135976+0000","last_active":"2024-09-13T11:40:18.047468+0000","last_peered":"2024-09-13T11:40:18.047468+0000","last_clean":"2024-09-13T11:40:18.047468+0000","last_became_active":"2024-09-13T11:40:12.105280+0000","last_became_peered":"2024-09-13T11:40:12.105280+0000","last_unstale":"2024-09-13T11:40:18.047468+0000","last_undegraded":"2024-09-13T11:40:18.047468+0000","last_fullsized":"2024-09-13T11:40:18.047468+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:20.791570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001476088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T11:40:14.138397+0000","last_change":"2024-09-13T11:40:14.138397+0000","last_active":"2024-09-13T11:40:14.138397+0000","last_peered":"2024-09-13T11:40:14.138397+0000","last_clean":"2024-09-13T11:40:14.138397+0000","last_became_active":"2024-09-13T11:40:12.113752+0000","last_became_peered":"2024-09-13T11:40:12.113752+0000","last_unstale":"2024-09-13T11:40:14.138397+0000","last_undegraded":"2024-09-13T11:40:14.138397+0000","last_fullsized":"2024-09-13T11:40:14.138397+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:50:52.508777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086960399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":113,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T11:40:18.047440+0000","last_change":"2024-09-13T11:40:05.394873+0000","last_active":"2024-09-13T11:40:18.047440+0000","last_peered":"2024-09-13T11:40:18.047440+0000","last_clean":"2024-09-13T11:40:18.047440+0000","last_became_active":"2024-09-13T11:40:05.394176+0000","last_became_peered":"2024-09-13T11:40:05.394176+0000","last_unstale":"2024-09-13T11:40:18.047440+0000","last_undegraded":"2024-09-13T11:40:18.047440+0000","last_fullsized":"2024-09-13T11:40:18.047440+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_clean_scrub_stamp":"2024-09-13T11:40:04.377013+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:00:31.498589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":584682,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":55773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":163840,"data_stored":51992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":580920,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:40:20.708 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-13T11:40:20.709 DEBUG:tasks.mon_thrash:Yielding 2024-09-13T11:40:20.709 INFO:teuthology.run_tasks:Running task exec... 2024-09-13T11:40:20.718 INFO:teuthology.task.exec:Executing custom commands... 2024-09-13T11:40:20.719 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T11:40:20.719 DEBUG:teuthology.orchestra.run.smithi039:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20' 2024-09-13T11:40:20.719 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-13T11:40:20.719 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726227620, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-13T11:40:20.719 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T11:40:20.720 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:40:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient: get_monmap_and_config 2024-09-13T11:40:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient: build_initial_monmap 2024-09-13T11:40:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient: monmap: 2024-09-13T11:40:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:20.833954+0000 2024-09-13T11:40:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:20.833954+0000 2024-09-13T11:40:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:20.833 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:20.833 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:20.833 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:20.833 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 1 Processor -- start 2024-09-13T11:40:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 1 -- start start 2024-09-13T11:40:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient: init 2024-09-13T11:40:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-13T11:40:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4fac134810 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fac137660 0x7f4fac139a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient(hunting): picked mon.noname-b con 0x7f4fac137660 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4fac139f90 0x7f4fac13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient(hunting): picked mon.noname-a con 0x7f4fac139f90 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fac09ce60 con 0x7f4fac134810 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f4fac0a0950 con 0x7f4fac139f90 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f4fac09eeb0 con 0x7f4fac137660 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fb1f19640 10 monclient(hunting): authenticate will time out at 1024.182754s 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4fac139f90 0x7f4fac13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4fac139f90 0x7f4fac13c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42450/0 (socket says 172.21.15.5:42450) 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 1 -- 172.21.15.5:0/2904815594 learned_addr learned my addr 172.21.15.5:0/2904815594 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 10 monclient(hunting): get_auth_request con 0x7f4fac139f90 auth_method 0 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4faaffd640 1 --2- 172.21.15.5:0/2904815594 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fac137660 0x7f4fac139a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4faaffd640 10 monclient(hunting): get_auth_request con 0x7f4fac137660 auth_method 0 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4faaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4faaffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4faaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4faaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4faaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4faaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 10 monclient(hunting): handle_auth_done global_id 4449 payload 274 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 1 -- 172.21.15.5:0/2904815594 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fac137660 msgr2=0x7f4fac139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 1 --2- 172.21.15.5:0/2904815594 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fac137660 0x7f4fac139a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 1 -- 172.21.15.5:0/2904815594 >> v1:172.21.15.5:6789/0 conn(0x7f4fac134810 legacy=0x7f4fac134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 1 monclient: found mon.noname-a 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4faa7fc640 1 -- 172.21.15.5:0/2904815594 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1456076566 0 0) 0x7f4fac09ce60 con 0x7f4fac134810 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4fabfff640 1 -- 172.21.15.5:0/2904815594 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fac09e1e0 con 0x7f4fac139f90 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4faa7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:40:20.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.831+0000 7f4faaffd640 1 --2- 172.21.15.5:0/2904815594 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fac137660 0x7f4fac139a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:20.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fabfff640 1 --2- 172.21.15.5:0/2904815594 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fac139f90 0x7f4fac13c380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400d150 tx=0x7f4fa401eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a83e5d7c8bf6c4b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 1 -- 172.21.15.5:0/2904815594 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa400b2d0 con 0x7f4fac139f90 2024-09-13T11:40:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 10 monclient: dump: 2024-09-13T11:40:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 10 monclient: _finish_auth 0 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 5 monclient: authenticate success, global_id 4449 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 1 -- 172.21.15.5:0/2904815594 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa400b470 con 0x7f4fac139f90 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 1 -- 172.21.15.5:0/2904815594 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40a79e0 con 0x7f4fac139f90 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 4 set_mon_vals no callback set 2024-09-13T11:40:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4faa7fc640 10 monclient: dump: 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient: shutdown 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2904815594 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fac139f90 msgr2=0x7f4fac13c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 --2- 172.21.15.5:0/2904815594 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fac139f90 0x7f4fac13c380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400d150 tx=0x7f4fa401eaf0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2904815594 shutdown_connections 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 --2- 172.21.15.5:0/2904815594 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fac139f90 0x7f4fac13c380 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 --2- 172.21.15.5:0/2904815594 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fac137660 0x7f4fac139a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2904815594 >> 172.21.15.5:0/2904815594 conn(0x7f4fac0a2930 msgr2=0x7f4fac0a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2904815594 shutdown_connections 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2904815594 wait complete. 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient: build_initial_monmap 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient: monmap: 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 Processor -- start 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 -- start start 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient: init 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:40:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fac134810 0x7f4fac073b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient(hunting): picked mon.noname-c con 0x7f4fac134810 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fac137660 0x7f4fac1cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient(hunting): picked mon.noname-a con 0x7f4fac137660 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fac139f90 0x7f4fac133b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient(hunting): picked mon.noname-b con 0x7f4fac139f90 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac0a16c0 con 0x7f4fac137660 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac0a1540 con 0x7f4fac139f90 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f4fac0a1840 con 0x7f4fac134810 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.835+0000 7f4fb1f19640 10 monclient(hunting): authenticate will time out at 1024.187973s 2024-09-13T11:40:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fac137660 0x7f4fac1cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fab7fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fac134810 0x7f4fac073b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fac137660 0x7f4fac1cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42458/0 (socket says 172.21.15.5:42458) 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fab7fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fac134810 0x7f4fac073b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47066/0 (socket says 172.21.15.5:47066) 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 1 -- 172.21.15.5:0/2813882578 learned_addr learned my addr 172.21.15.5:0/2813882578 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 10 monclient(hunting): get_auth_request con 0x7f4fac137660 auth_method 0 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fab7fe640 10 monclient(hunting): get_auth_request con 0x7f4fac134810 auth_method 0 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fabfff640 1 --2- 172.21.15.5:0/2813882578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fac139f90 0x7f4fac133b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fabfff640 10 monclient(hunting): get_auth_request con 0x7f4fac139f90 auth_method 0 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fabfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 10 monclient(hunting): handle_auth_done global_id 4455 payload 824 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 1 -- 172.21.15.5:0/2813882578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fac134810 msgr2=0x7f4fac073b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 1 --2- 172.21.15.5:0/2813882578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fac134810 0x7f4fac073b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 1 -- 172.21.15.5:0/2813882578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fac139f90 msgr2=0x7f4fac133b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:20.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 1 --2- 172.21.15.5:0/2813882578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fac139f90 0x7f4fac133b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 10 monclient: _finish_hunting 0 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 1 monclient: found mon.noname-a 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 1 -- 172.21.15.5:0/2813882578 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fac072ff0 con 0x7f4fac137660 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4faaffd640 1 --2- 172.21.15.5:0/2813882578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fac137660 0x7f4fac1cb170 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000ec10 tx=0x7f4fa000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 1 -- 172.21.15.5:0/2813882578 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0017070 con 0x7f4fac137660 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 10 monclient: dump: 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 10 monclient: _finish_auth 0 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 1 -- 172.21.15.5:0/2813882578 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa000cbd0 con 0x7f4fac137660 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fb1f19640 5 monclient: authenticate success, global_id 4455 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fb1f19640 10 monclient: _renew_subs 2024-09-13T11:40:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fb1f19640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2813882578 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fac06dbf0 con 0x7f4fac137660 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 1 -- 172.21.15.5:0/2813882578 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0013470 con 0x7f4fac137660 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 10 monclient: dump: 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:20.854 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:20.855 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:20.855 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:20.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fb1f19640 10 monclient: _renew_subs 2024-09-13T11:40:20.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fb1f19640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:20.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2813882578 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fac1cb6b0 con 0x7f4fac137660 2024-09-13T11:40:20.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fa9ffb640 4 set_mon_vals no callback set 2024-09-13T11:40:20.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fb1f19640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:20.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fb1f19640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:20.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fb1f19640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:20.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2813882578 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fac0a1ca0 con 0x7f4fac137660 2024-09-13T11:40:20.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 1 -- 172.21.15.5:0/2813882578 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f4fa001a070 con 0x7f4fac137660 2024-09-13T11:40:20.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 1 --2- 172.21.15.5:0/2813882578 >> v2:172.21.15.39:6812/1340838781 conn(0x7f4f9803b2a0 0x7f4f9803d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:20.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4f93fff640 1 -- 172.21.15.5:0/2813882578 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f4fa004ebd0 con 0x7f4fac137660 2024-09-13T11:40:20.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fab7fe640 1 --2- 172.21.15.5:0/2813882578 >> v2:172.21.15.39:6812/1340838781 conn(0x7f4f9803b2a0 0x7f4f9803d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:20.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fab7fe640 10 monclient: get_auth_request con 0x7f4f9803b2a0 auth_method 0 2024-09-13T11:40:20.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.839+0000 7f4fab7fe640 1 --2- 172.21.15.5:0/2813882578 >> v2:172.21.15.39:6812/1340838781 conn(0x7f4f9803b2a0 0x7f4f9803d760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c000f70 tx=0x7f4f9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:20.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.843+0000 7f4f93fff640 1 -- 172.21.15.5:0/2813882578 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f4fa0013b30 con 0x7f4fac137660 2024-09-13T11:40:20.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.843+0000 7f4f93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:20.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:20.843+0000 7f4f93fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:21.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.071+0000 7f4fb1f19640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:40:21.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.071+0000 7f4fb1f19640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:40:21.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.071+0000 7f4fb1f19640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:21.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.071+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2813882578 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4fac069ff0 con 0x7f4fac137660 2024-09-13T11:40:21.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.071+0000 7f4f93fff640 1 -- 172.21.15.5:0/2813882578 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1548 (secure 0 0 0) 0x7f4fa0013610 con 0x7f4fac137660 2024-09-13T11:40:21.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.071+0000 7f4f93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:40:21.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.071+0000 7f4f93fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:21.074 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","modified":"2024-09-13T11:39:20.750174Z","created":"2024-09-13T11:39:20.750174Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:40:21.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2813882578 >> v2:172.21.15.39:6812/1340838781 conn(0x7f4f9803b2a0 msgr2=0x7f4f9803d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 1 --2- 172.21.15.5:0/2813882578 >> v2:172.21.15.39:6812/1340838781 conn(0x7f4f9803b2a0 0x7f4f9803d760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c000f70 tx=0x7f4f9c009040 comp rx=0 tx=0).stop 2024-09-13T11:40:21.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 10 monclient: shutdown 2024-09-13T11:40:21.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:21.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2813882578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fac137660 msgr2=0x7f4fac1cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 1 --2- 172.21.15.5:0/2813882578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fac137660 0x7f4fac1cb170 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000ec10 tx=0x7f4fa000c520 comp rx=0 tx=0).stop 2024-09-13T11:40:21.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2813882578 shutdown_connections 2024-09-13T11:40:21.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 1 --2- 172.21.15.5:0/2813882578 >> v2:172.21.15.39:6812/1340838781 conn(0x7f4f9803b2a0 0x7f4f9803d760 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 1 --2- 172.21.15.5:0/2813882578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fac139f90 0x7f4fac133b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 1 --2- 172.21.15.5:0/2813882578 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4fac137660 0x7f4fac1cb170 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 1 --2- 172.21.15.5:0/2813882578 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4fac134810 0x7f4fac073b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2813882578 >> 172.21.15.5:0/2813882578 conn(0x7f4fac0a2930 msgr2=0x7f4fac136e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:21.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2813882578 shutdown_connections 2024-09-13T11:40:21.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.075+0000 7f4fb1f19640 1 -- 172.21.15.5:0/2813882578 wait complete. 2024-09-13T11:40:21.093 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T11:40:21.093 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-13T11:40:21.093 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T11:40:21.093 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T11:40:21.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient: get_monmap_and_config 2024-09-13T11:40:21.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient: build_initial_monmap 2024-09-13T11:40:21.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient: monmap: 2024-09-13T11:40:21.209 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:21.209 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.209 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:21.211236+0000 2024-09-13T11:40:21.209 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:21.211236+0000 2024-09-13T11:40:21.210 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:21.210 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.210 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:21.210 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:21.210 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:21.210 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:21.210 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:21.210 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 1 Processor -- start 2024-09-13T11:40:21.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 1 -- start start 2024-09-13T11:40:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient: init 2024-09-13T11:40:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-13T11:40:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2c10137930 0x7f2c1009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient(hunting): picked mon.noname-a con 0x7f2c10137930 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c10137d10 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2c1013cd80 0x7f2c1013f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient(hunting): picked mon.noname-c con 0x7f2c1013cd80 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c100a5a80 con 0x7f2c10137d10 2024-09-13T11:40:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f2c100a5fe0 con 0x7f2c10137930 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f2c1009c540 con 0x7f2c1013cd80 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.207+0000 7f2c150de640 10 monclient(hunting): authenticate will time out at 1024.559920s 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2c10137930 0x7f2c1009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2c10137930 0x7f2c1009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42466/0 (socket says 172.21.15.5:42466) 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 1 -- 172.21.15.5:0/2530310081 learned_addr learned my addr 172.21.15.5:0/2530310081 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 10 monclient(hunting): get_auth_request con 0x7f2c10137930 auth_method 0 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0f59e640 1 --2- 172.21.15.5:0/2530310081 >> v2:172.21.15.39:3300/0 conn(0x7f2c1013cd80 0x7f2c1013f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 10 monclient(hunting): handle_auth_done global_id 4458 payload 274 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 1 -- 172.21.15.5:0/2530310081 >> v2:172.21.15.39:3300/0 conn(0x7f2c1013cd80 msgr2=0x7f2c1013f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 1 --2- 172.21.15.5:0/2530310081 >> v2:172.21.15.39:3300/0 conn(0x7f2c1013cd80 0x7f2c1013f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 1 -- 172.21.15.5:0/2530310081 >> v1:172.21.15.39:6789/0 conn(0x7f2c10137d10 legacy=0x7f2c101380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 10 monclient: _finish_hunting 0 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 1 monclient: found mon.noname-a 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 1 -- 172.21.15.5:0/2530310081 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1009bef0 con 0x7f2c10137930 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0ed9d640 1 --2- 172.21.15.5:0/2530310081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c10137930 0x7f2c1009cc90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2c10138800 tx=0x7f2c0001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32b0cd7920765e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 1 -- 172.21.15.5:0/2530310081 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c000b0070 con 0x7f2c10137930 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 10 monclient: dump: 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 10 monclient: _finish_auth 0 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 10 monclient: _check_auth_tickets 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 1 -- 172.21.15.5:0/2530310081 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c000ab600 con 0x7f2c10137930 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 5 monclient: authenticate success, global_id 4458 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 1 -- 172.21.15.5:0/2530310081 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c000ab9d0 con 0x7f2c10137930 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 4 set_mon_vals no callback set 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c0dd9b640 10 monclient: dump: 2024-09-13T11:40:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.220 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 10 monclient: shutdown 2024-09-13T11:40:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 1 -- 172.21.15.5:0/2530310081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c10137930 msgr2=0x7f2c1009cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 1 --2- 172.21.15.5:0/2530310081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c10137930 0x7f2c1009cc90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2c10138800 tx=0x7f2c0001c3e0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 1 -- 172.21.15.5:0/2530310081 shutdown_connections 2024-09-13T11:40:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 1 --2- 172.21.15.5:0/2530310081 >> v2:172.21.15.39:3300/0 conn(0x7f2c1013cd80 0x7f2c1013f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 1 --2- 172.21.15.5:0/2530310081 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c10137930 0x7f2c1009cc90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 1 -- 172.21.15.5:0/2530310081 >> 172.21.15.5:0/2530310081 conn(0x7f2c100a6c70 msgr2=0x7f2c100a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 1 -- 172.21.15.5:0/2530310081 shutdown_connections 2024-09-13T11:40:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 1 -- 172.21.15.5:0/2530310081 wait complete. 2024-09-13T11:40:21.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 10 monclient: build_initial_monmap 2024-09-13T11:40:21.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 10 monclient: monmap: 2024-09-13T11:40:21.227 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:21.227 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:21.227 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.211+0000 7f2c150de640 1 Processor -- start 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 1 -- start start 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient: init 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c10137930 0x7f2c10136790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient(hunting): picked mon.noname-a con 0x7f2c10137930 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c10137d10 0x7f2c10136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient(hunting): picked mon.noname-c con 0x7f2c10137d10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c1013cd80 0x7f2c10139b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient(hunting): picked mon.noname-b con 0x7f2c1013cd80 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2c10142320 con 0x7f2c10137930 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2c101421a0 con 0x7f2c1013cd80 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f2c101424a0 con 0x7f2c10137d10 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient(hunting): authenticate will time out at 1024.564485s 2024-09-13T11:40:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c10137930 0x7f2c10136790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c10137d10 0x7f2c10136cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c10137930 0x7f2c10136790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42468/0 (socket says 172.21.15.5:42468) 2024-09-13T11:40:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 1 -- 172.21.15.5:0/1065441755 learned_addr learned my addr 172.21.15.5:0/1065441755 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 10 monclient(hunting): get_auth_request con 0x7f2c10137930 auth_method 0 2024-09-13T11:40:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): get_auth_request con 0x7f2c10137d10 auth_method 0 2024-09-13T11:40:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 10 monclient(hunting): handle_auth_done global_id 4461 payload 824 2024-09-13T11:40:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 1 -- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c10137d10 msgr2=0x7f2c10136cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c10137d10 0x7f2c10136cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 1 -- 172.21.15.5:0/1065441755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c1013cd80 msgr2=0x7f2c10139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0f59e640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c1013cd80 0x7f2c10139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c1013cd80 0x7f2c10139b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 10 monclient: _finish_hunting 0 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 1 monclient: found mon.noname-a 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 1 -- 172.21.15.5:0/1065441755 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c10121020 con 0x7f2c10137930 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0f59e640 10 monclient: get_auth_request con 0x7f2c1013cd80 auth_method 0 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0f59e640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c1013cd80 0x7f2c10139b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0ed9d640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c10137930 0x7f2c10136790 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2c000ae040 tx=0x7f2c00019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c000b0040 con 0x7f2c10137930 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: dump: 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c000b9070 con 0x7f2c10137930 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c000b48a0 con 0x7f2c10137930 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: dump: 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.234 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 5 monclient: authenticate success, global_id 4461 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient: _renew_subs 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 1 -- 172.21.15.5:0/1065441755 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c10121300 con 0x7f2c10137930 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0d59a640 4 set_mon_vals no callback set 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient: _renew_subs 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 1 -- 172.21.15.5:0/1065441755 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c100a1360 con 0x7f2c10137930 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a0d0 0x7f2c1013a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c150de640 1 -- 172.21.15.5:0/1065441755 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1009cc90 con 0x7f2c1013a0d0 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a0d0 0x7f2c1013a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient: get_auth_request con 0x7f2c1013a0d0 auth_method 0 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f2c000ab4c0 con 0x7f2c10137930 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 1 -- 172.21.15.5:0/1065441755 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c10120da0 con 0x7f2c1013a0d0 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0e59c640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a0d0 0x7f2c1013a4b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c10121230 tx=0x7f2c0400fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 1 --2- 172.21.15.5:0/1065441755 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2bfc03b2a0 0x7f2bfc03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (secure 0 0 0) 0x7f2c000e8bd0 con 0x7f2c10137930 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c04018020 con 0x7f2c1013a0d0 2024-09-13T11:40:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: dump: 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0f59e640 1 --2- 172.21.15.5:0/1065441755 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2bfc03b2a0 0x7f2bfc03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0f59e640 10 monclient: get_auth_request con 0x7f2bfc03b2a0 auth_method 0 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2c0400d770 con 0x7f2c1013a0d0 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a0d0 msgr2=0x7f2c1013a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a0d0 0x7f2c1013a4b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c10121230 tx=0x7f2c0400fc00 comp rx=0 tx=0).stop 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a0d0 msgr2=0x7f2c1013a4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2bf77fe640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a0d0 0x7f2c1013a4b0 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c10121230 tx=0x7f2c0400fc00 comp rx=0 tx=0).stop 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.215+0000 7f2c0f59e640 1 --2- 172.21.15.5:0/1065441755 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2bfc03b2a0 0x7f2bfc03d760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2c08007eb0 tx=0x7f2c08007400 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a9f0 0x7f2c100a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 1 -- 172.21.15.5:0/1065441755 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1006cbe0 con 0x7f2c1013a9f0 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0e59c640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a9f0 0x7f2c100a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0e59c640 10 monclient: get_auth_request con 0x7f2c1013a9f0 auth_method 0 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0e59c640 10 monclient(hunting): handle_auth_done global_id 4467 payload 824 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0e59c640 1 -- 172.21.15.5:0/1065441755 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2c1013bd90 con 0x7f2c1013a9f0 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0e59c640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a9f0 0x7f2c100a1530 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2c04019000 tx=0x7f2c04003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c04018030 con 0x7f2c1013a9f0 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2bf77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2bf77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2bf77fe640 10 monclient: dump: 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7f2c0401fd30 con 0x7f2c1013a9f0 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2bf77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2bf77fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a9f0 msgr2=0x7f2c100a1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2bf77fe640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a9f0 0x7f2c100a1530 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2c04019000 tx=0x7f2c04003770 comp rx=0 tx=0).stop 2024-09-13T11:40:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2bf77fe640 1 -- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a9f0 msgr2=0x7f2c100a1530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:40:21.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2bf77fe640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c1013a9f0 0x7f2c100a1530 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2c04019000 tx=0x7f2c04003770 comp rx=0 tx=0).stop 2024-09-13T11:40:21.241 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:40:21.241 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-13T11:40:21.241 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-13T11:40:21.241 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-13T11:40:21.241 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 8, 2024-09-13T11:40:21.241 INFO:teuthology.orchestra.run.smithi005.stdout: "uptime": 33195, 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 27, 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2450521708007866372", 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:21.242 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:21.243 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-13T11:40:21.244 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "59046471-79a9-4d90-9616-da0b446053a7", 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 19, 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "squid", 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 1, 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:21.245 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:21.246 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:21.246 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:21.246 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:21.246 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:21.246 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:21.246 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:21.246 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:21.246 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:21.246 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.246 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-13T11:40:21.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.246 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:21.247 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-13T11:40:21.248 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.248 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.248 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.248 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-13T11:40:21.248 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-13T11:40:21.248 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:21.248 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:21.248 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:21.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.249 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-13T11:40:21.249 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-13T11:40:21.249 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:21.249 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:21.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.249 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:21.249 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:3300", 2024-09-13T11:40:21.249 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.249 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789", 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789/0", 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.39:6789/0", 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-13T11:40:21.250 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:40:21.252 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-13T11:40:21.253 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.254 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.254 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-13T11:40:21.254 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:40:21.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 1 -- 172.21.15.5:0/1065441755 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2bfc03b2a0 msgr2=0x7f2bfc03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 1 --2- 172.21.15.5:0/1065441755 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2bfc03b2a0 0x7f2bfc03d760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2c08007eb0 tx=0x7f2c08007400 comp rx=0 tx=0).stop 2024-09-13T11:40:21.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 10 monclient: shutdown 2024-09-13T11:40:21.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:21.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 1 -- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c10137930 msgr2=0x7f2c10136790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 1 --2- 172.21.15.5:0/1065441755 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c10137930 0x7f2c10136790 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2c000ae040 tx=0x7f2c00019180 comp rx=0 tx=0).stop 2024-09-13T11:40:21.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0f59e640 1 -- 172.21.15.5:0/1065441755 reap_dead start 2024-09-13T11:40:21.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c0f59e640 1 -- 172.21.15.5:0/1065441755 reap_dead start 2024-09-13T11:40:21.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 1 -- 172.21.15.5:0/1065441755 shutdown_connections 2024-09-13T11:40:21.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 1 -- 172.21.15.5:0/1065441755 >> 172.21.15.5:0/1065441755 conn(0x7f2c100a6c70 msgr2=0x7f2c1009fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:21.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 1 -- 172.21.15.5:0/1065441755 shutdown_connections 2024-09-13T11:40:21.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.223+0000 7f2c150de640 1 -- 172.21.15.5:0/1065441755 wait complete. 2024-09-13T11:40:21.257 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:40:21.257 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T11:40:21.257 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T11:40:21.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.363+0000 7f782f30a640 10 monclient: get_monmap_and_config 2024-09-13T11:40:21.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.363+0000 7f782f30a640 10 monclient: build_initial_monmap 2024-09-13T11:40:21.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.363+0000 7f782f30a640 10 monclient: monmap: 2024-09-13T11:40:21.367 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:21.367 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.367 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:21.368736+0000 2024-09-13T11:40:21.367 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:21.368736+0000 2024-09-13T11:40:21.367 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:21.367 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.367 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:21.367 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:21.367 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:21.367 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:21.368 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:21.368 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 1 Processor -- start 2024-09-13T11:40:21.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 1 -- start start 2024-09-13T11:40:21.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 10 monclient: init 2024-09-13T11:40:21.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:21.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-13T11:40:21.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7828137930 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:21.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7828138420 0x7f782809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 10 monclient(hunting): picked mon.noname-c con 0x7f7828138420 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:21.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f782813cd80 0x7f782813f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 10 monclient(hunting): picked mon.noname-a con 0x7f782813cd80 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:21.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78280a5a80 con 0x7f7828137930 2024-09-13T11:40:21.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f78280a5fe0 con 0x7f782813cd80 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f782809c540 con 0x7f7828138420 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782f30a640 10 monclient(hunting): authenticate will time out at 1024.717605s 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f782813cd80 0x7f782813f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d0a6640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f7828137930 0x7f7828137d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:56420/0 (socket says 172.21.15.5:56420) 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f782813cd80 0x7f782813f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42486/0 (socket says 172.21.15.5:42486) 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d0a6640 1 -- 172.21.15.5:0/2590831876 learned_addr learned my addr 172.21.15.5:0/2590831876 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 10 monclient(hunting): get_auth_request con 0x7f782813cd80 auth_method 0 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782c8a5640 1 --2- 172.21.15.5:0/2590831876 >> v2:172.21.15.39:3300/0 conn(0x7f7828138420 0x7f782809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782c8a5640 10 monclient(hunting): get_auth_request con 0x7f7828138420 auth_method 0 2024-09-13T11:40:21.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782c8a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782c8a5640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782c8a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 10 monclient(hunting): handle_auth_done global_id 4470 payload 274 2024-09-13T11:40:21.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 1 -- 172.21.15.5:0/2590831876 >> v2:172.21.15.39:3300/0 conn(0x7f7828138420 msgr2=0x7f782809cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:21.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 1 --2- 172.21.15.5:0/2590831876 >> v2:172.21.15.39:3300/0 conn(0x7f7828138420 0x7f782809cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 1 -- 172.21.15.5:0/2590831876 >> v1:172.21.15.5:6789/0 conn(0x7f7828137930 legacy=0x7f7828137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 10 monclient: _finish_hunting 0 2024-09-13T11:40:21.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 1 monclient: found mon.noname-a 2024-09-13T11:40:21.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f7827fff640 1 -- 172.21.15.5:0/2590831876 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2593751867 0 0) 0x7f78280a5a80 con 0x7f7828137930 2024-09-13T11:40:21.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:21.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:21.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782d8a7640 1 -- 172.21.15.5:0/2590831876 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782809bef0 con 0x7f782813cd80 2024-09-13T11:40:21.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f782c8a5640 1 --2- 172.21.15.5:0/2590831876 >> v2:172.21.15.39:3300/0 conn(0x7f7828138420 0x7f782809cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:40:21.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.367+0000 7f7827fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:40:21.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 1 --2- 172.21.15.5:0/2590831876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f782813cd80 0x7f782813f170 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7828070bd0 tx=0x7f781801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc4ef16b7465139 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 1 -- 172.21.15.5:0/2590831876 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78180b2070 con 0x7f782813cd80 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 10 monclient: dump: 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 10 monclient: _finish_auth 0 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 5 monclient: authenticate success, global_id 4470 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 1 -- 172.21.15.5:0/2590831876 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78180a57c0 con 0x7f782813cd80 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 4 set_mon_vals no callback set 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 1 -- 172.21.15.5:0/2590831876 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78180a5b90 con 0x7f782813cd80 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7827fff640 10 monclient: dump: 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.382 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient: shutdown 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 -- 172.21.15.5:0/2590831876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f782813cd80 msgr2=0x7f782813f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 --2- 172.21.15.5:0/2590831876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f782813cd80 0x7f782813f170 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7828070bd0 tx=0x7f781801c7f0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 -- 172.21.15.5:0/2590831876 shutdown_connections 2024-09-13T11:40:21.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 --2- 172.21.15.5:0/2590831876 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f782813cd80 0x7f782813f170 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 --2- 172.21.15.5:0/2590831876 >> v2:172.21.15.39:3300/0 conn(0x7f7828138420 0x7f782809cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 -- 172.21.15.5:0/2590831876 >> 172.21.15.5:0/2590831876 conn(0x7f78280a6c70 msgr2=0x7f78280a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 -- 172.21.15.5:0/2590831876 shutdown_connections 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 -- 172.21.15.5:0/2590831876 wait complete. 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient: build_initial_monmap 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient: monmap: 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.384 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 Processor -- start 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 -- start start 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient: init 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7828137930 0x7f782813beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient(hunting): picked mon.noname-a con 0x7f7828137930 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7828138420 0x7f7828136770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient(hunting): picked mon.noname-b con 0x7f7828138420 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f782813cd80 0x7f78281398e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient(hunting): picked mon.noname-c con 0x7f782813cd80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:21.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7828142350 con 0x7f7828137930 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f78281421d0 con 0x7f7828138420 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f78281424d0 con 0x7f782813cd80 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient(hunting): authenticate will time out at 1024.722452s 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d0a6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7828137930 0x7f782813beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d0a6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7828137930 0x7f782813beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42494/0 (socket says 172.21.15.5:42494) 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d0a6640 1 -- 172.21.15.5:0/4095104713 learned_addr learned my addr 172.21.15.5:0/4095104713 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d0a6640 10 monclient(hunting): get_auth_request con 0x7f7828137930 auth_method 0 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d0a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d0a6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d0a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f782813cd80 0x7f78281398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 10 monclient(hunting): get_auth_request con 0x7f782813cd80 auth_method 0 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d0a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d0a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d0a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782c8a5640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7828138420 0x7f7828136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782c8a5640 10 monclient(hunting): get_auth_request con 0x7f7828138420 auth_method 0 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782c8a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782c8a5640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782c8a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 10 monclient(hunting): handle_auth_done global_id 4346 payload 824 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 1 -- 172.21.15.5:0/4095104713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7828138420 msgr2=0x7f7828136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7828138420 0x7f7828136770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 1 -- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7828137930 msgr2=0x7f782813beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7828137930 0x7f782813beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 10 monclient: _finish_hunting 0 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 1 monclient: found mon.noname-c 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 1 -- 172.21.15.5:0/4095104713 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782813c840 con 0x7f782813cd80 2024-09-13T11:40:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d0a6640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7828137930 0x7f782813beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782d8a7640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f782813cd80 0x7f78281398e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f78180ac000 tx=0x7f7818019590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78180b2050 con 0x7f782813cd80 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 10 monclient: dump: 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78180be070 con 0x7f782813cd80 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78180ada90 con 0x7f782813cd80 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 5 monclient: authenticate success, global_id 4346 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient: _renew_subs 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 -- 172.21.15.5:0/4095104713 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7828121100 con 0x7f782813cd80 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f78277fe640 4 set_mon_vals no callback set 2024-09-13T11:40:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f7825ffb640 10 monclient: dump: 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient: _renew_subs 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 1 -- 172.21.15.5:0/4095104713 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f78280a1360 con 0x7f782813cd80 2024-09-13T11:40:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.371+0000 7f782f30a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782f30a640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7828139e20 0x7f782813a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782f30a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782f30a640 1 -- 172.21.15.5:0/4095104713 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f782813a740 con 0x7f7828139e20 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f78180a5640 con 0x7f782813cd80 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782d0a6640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7828139e20 0x7f782813a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782d0a6640 10 monclient: get_auth_request con 0x7f7828139e20 auth_method 0 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782d0a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782d0a6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782d0a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782d0a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782d0a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782d0a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 1 --2- 172.21.15.5:0/4095104713 >> v2:172.21.15.39:6812/1340838781 conn(0x7f781403b2a0 0x7f781403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782d0a6640 10 monclient(hunting): handle_auth_done global_id 4349 payload 824 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782d0a6640 1 -- 172.21.15.5:0/4095104713 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f782809cc90 con 0x7f7828139e20 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (secure 0 0 0) 0x7f78180e9f30 con 0x7f782813cd80 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782d0a6640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7828139e20 0x7f782813a200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7820004a20 tx=0x7f7820001e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782c8a5640 1 --2- 172.21.15.5:0/4095104713 >> v2:172.21.15.39:6812/1340838781 conn(0x7f781403b2a0 0x7f781403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782000fa40 con 0x7f7828139e20 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 10 monclient: dump: 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.394 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.394 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.394 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.394 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.395 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782c8a5640 10 monclient: get_auth_request con 0x7f781403b2a0 auth_method 0 2024-09-13T11:40:21.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7820018070 con 0x7f7828139e20 2024-09-13T11:40:21.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:21.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:21.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7828139e20 msgr2=0x7f782813a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7828139e20 0x7f782813a200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7820004a20 tx=0x7f7820001e80 comp rx=0 tx=0).stop 2024-09-13T11:40:21.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7828139e20 msgr2=0x7f782813a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:40:21.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f7825ffb640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7828139e20 0x7f782813a200 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7820004a20 tx=0x7f7820001e80 comp rx=0 tx=0).stop 2024-09-13T11:40:21.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.375+0000 7f782c8a5640 1 --2- 172.21.15.5:0/4095104713 >> v2:172.21.15.39:6812/1340838781 conn(0x7f781403b2a0 0x7f781403d760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f78280a0ef0 tx=0x7f781c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782f30a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:40:21.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782f30a640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f782809db50 0x7f782809df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782f30a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782f30a640 1 -- 172.21.15.5:0/4095104713 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f782806cbe0 con 0x7f782809db50 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782d0a6640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f782809db50 0x7f782809df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782d0a6640 10 monclient: get_auth_request con 0x7f782809db50 auth_method 0 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782d0a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782d0a6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782d0a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782d0a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782d0a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782d0a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782d0a6640 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782d0a6640 1 -- 172.21.15.5:0/4095104713 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7828120d80 con 0x7f782809db50 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f782d0a6640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f782809db50 0x7f782809df30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7820004a20 tx=0x7f7820000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78200138c0 con 0x7f782809db50 2024-09-13T11:40:21.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f7825ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f7825ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f7825ffb640 10 monclient: dump: 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4699 (secure 0 0 0) 0x7f78200138c0 con 0x7f782809db50 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f7825ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:40:21.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f7825ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:21.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f782809db50 msgr2=0x7f782809df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f7825ffb640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f782809db50 0x7f782809df30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7820004a20 tx=0x7f7820000ee0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f7825ffb640 1 -- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f782809db50 msgr2=0x7f782809df30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:40:21.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.379+0000 7f7825ffb640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f782809db50 0x7f782809df30 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7820004a20 tx=0x7f7820000ee0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.400 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:40:21.400 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-13T11:40:21.400 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-13T11:40:21.400 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-13T11:40:21.400 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 8, 2024-09-13T11:40:21.400 INFO:teuthology.orchestra.run.smithi005.stdout: "uptime": 33351, 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 28, 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2450521708007866372", 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:21.401 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:21.402 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:21.402 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:21.402 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:21.402 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:21.402 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:21.402 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:21.402 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.402 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:40:21.402 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-13T11:40:21.404 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:21.404 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:21.404 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:21.404 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:21.404 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:21.404 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "59046471-79a9-4d90-9616-da0b446053a7", 2024-09-13T11:40:21.405 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 19, 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "squid", 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 1, 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:21.406 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-13T11:40:21.407 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.408 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:21.409 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:3300", 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789", 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789/0", 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.39:6789/0", 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:21.410 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-13T11:40:21.411 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-13T11:40:21.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.413 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:21.414 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:21.414 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-13T11:40:21.414 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.414 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.414 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.414 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-13T11:40:21.414 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:40:21.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.383+0000 7f782f30a640 1 -- 172.21.15.5:0/4095104713 >> v2:172.21.15.39:6812/1340838781 conn(0x7f781403b2a0 msgr2=0x7f781403d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.383+0000 7f782f30a640 1 --2- 172.21.15.5:0/4095104713 >> v2:172.21.15.39:6812/1340838781 conn(0x7f781403b2a0 0x7f781403d760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f78280a0ef0 tx=0x7f781c008040 comp rx=0 tx=0).stop 2024-09-13T11:40:21.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.383+0000 7f782f30a640 10 monclient: shutdown 2024-09-13T11:40:21.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.383+0000 7f782f30a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:21.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.383+0000 7f782f30a640 1 -- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f782813cd80 msgr2=0x7f78281398e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.383+0000 7f782f30a640 1 --2- 172.21.15.5:0/4095104713 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f782813cd80 0x7f78281398e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f78180ac000 tx=0x7f7818019590 comp rx=0 tx=0).stop 2024-09-13T11:40:21.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.383+0000 7f782d8a7640 1 -- 172.21.15.5:0/4095104713 reap_dead start 2024-09-13T11:40:21.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.383+0000 7f782d8a7640 1 -- 172.21.15.5:0/4095104713 reap_dead start 2024-09-13T11:40:21.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.383+0000 7f782f30a640 1 -- 172.21.15.5:0/4095104713 shutdown_connections 2024-09-13T11:40:21.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.383+0000 7f782f30a640 1 -- 172.21.15.5:0/4095104713 >> 172.21.15.5:0/4095104713 conn(0x7f78280a6c70 msgr2=0x7f782813dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:21.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.383+0000 7f782f30a640 1 -- 172.21.15.5:0/4095104713 shutdown_connections 2024-09-13T11:40:21.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.383+0000 7f782f30a640 1 -- 172.21.15.5:0/4095104713 wait complete. 2024-09-13T11:40:21.417 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:40:21.417 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T11:40:21.417 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T11:40:21.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.523+0000 7fe4319cb640 10 monclient: get_monmap_and_config 2024-09-13T11:40:21.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.523+0000 7fe4319cb640 10 monclient: build_initial_monmap 2024-09-13T11:40:21.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.523+0000 7fe4319cb640 10 monclient: monmap: 2024-09-13T11:40:21.526 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:21.526 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.527 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:21.528366+0000 2024-09-13T11:40:21.527 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:21.528366+0000 2024-09-13T11:40:21.527 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:21.527 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.527 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:21.527 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:21.527 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:21.527 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:21.527 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:21.527 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 1 Processor -- start 2024-09-13T11:40:21.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 1 -- start start 2024-09-13T11:40:21.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 10 monclient: init 2024-09-13T11:40:21.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:21.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-13T11:40:21.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137930 0x7fe42c09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 10 monclient(hunting): picked mon.noname-b con 0x7fe42c137930 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:21.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe42c137d10 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:21.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe42c13cd80 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:21.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe42c0a5a80 con 0x7fe42c13cd80 2024-09-13T11:40:21.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe42c09bef0 con 0x7fe42c137d10 2024-09-13T11:40:21.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fe42c0a5fe0 con 0x7fe42c137930 2024-09-13T11:40:21.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:21.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe4319cb640 10 monclient(hunting): authenticate will time out at 1024.877182s 2024-09-13T11:40:21.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137930 0x7fe42c09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137930 0x7fe42c09cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47098/0 (socket says 172.21.15.5:47098) 2024-09-13T11:40:21.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 1 -- 172.21.15.5:0/365197297 learned_addr learned my addr 172.21.15.5:0/365197297 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:21.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 10 monclient(hunting): get_auth_request con 0x7fe42c137930 auth_method 0 2024-09-13T11:40:21.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 10 monclient(hunting): handle_auth_done global_id 4355 payload 274 2024-09-13T11:40:21.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 1 -- 172.21.15.5:0/365197297 >> v1:172.21.15.39:6789/0 conn(0x7fe42c137d10 legacy=0x7fe42c1380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:21.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 1 -- 172.21.15.5:0/365197297 >> v1:172.21.15.5:6789/0 conn(0x7fe42c13cd80 legacy=0x7fe42c13f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 10 monclient: _finish_hunting 0 2024-09-13T11:40:21.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 1 monclient: found mon.noname-b 2024-09-13T11:40:21.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:21.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:40:21.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 1 -- 172.21.15.5:0/365197297 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42c09c1d0 con 0x7fe42c137930 2024-09-13T11:40:21.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe42affd640 1 --2- 172.21.15.5:0/365197297 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137930 0x7fe42c09cc90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe41c0098a0 tx=0x7fe41c01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4af73d9fb001c219 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe429ffb640 1 -- 172.21.15.5:0/365197297 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c0b0070 con 0x7fe42c137930 2024-09-13T11:40:21.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe429ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe429ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:21.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe429ffb640 10 monclient: dump: 2024-09-13T11:40:21.542 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.542 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.542 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.542 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.542 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.542 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.542 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe429ffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe429ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe429ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe429ffb640 1 -- 172.21.15.5:0/365197297 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe41c0ab600 con 0x7fe42c137930 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe429ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe429ffb640 1 -- 172.21.15.5:0/365197297 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c0ab9d0 con 0x7fe42c137930 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.527+0000 7fe429ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe429ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe429ffb640 10 monclient: dump: 2024-09-13T11:40:21.543 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.544 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.544 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.544 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.544 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.544 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.544 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.544 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.544 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.544 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 5 monclient: authenticate success, global_id 4355 2024-09-13T11:40:21.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:21.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 4 set_mon_vals no callback set 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient: shutdown 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 -- 172.21.15.5:0/365197297 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137930 msgr2=0x7fe42c09cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 --2- 172.21.15.5:0/365197297 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137930 0x7fe42c09cc90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe41c0098a0 tx=0x7fe41c01c440 comp rx=0 tx=0).stop 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 -- 172.21.15.5:0/365197297 shutdown_connections 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 --2- 172.21.15.5:0/365197297 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137930 0x7fe42c09cc90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 -- 172.21.15.5:0/365197297 >> 172.21.15.5:0/365197297 conn(0x7fe42c0a6c70 msgr2=0x7fe42c0a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 -- 172.21.15.5:0/365197297 shutdown_connections 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 -- 172.21.15.5:0/365197297 wait complete. 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient: build_initial_monmap 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient: monmap: 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:21.545 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 Processor -- start 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 -- start start 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient: init 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:40:21.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe42c137930 0x7fe42c1cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient(hunting): picked mon.noname-a con 0x7fe42c137930 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137d10 0x7fe42c136920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient(hunting): picked mon.noname-c con 0x7fe42c137d10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c13cd80 0x7fe42c138d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient(hunting): picked mon.noname-b con 0x7fe42c13cd80 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c1438a0 con 0x7fe42c137930 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c143720 con 0x7fe42c13cd80 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fe42c143a20 con 0x7fe42c137d10 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient(hunting): authenticate will time out at 1024.882286s 2024-09-13T11:40:21.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe42c137930 0x7fe42c1cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42a7fc640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137d10 0x7fe42c136920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe42c137930 0x7fe42c1cfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42506/0 (socket says 172.21.15.5:42506) 2024-09-13T11:40:21.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42a7fc640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137d10 0x7fe42c136920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47106/0 (socket says 172.21.15.5:47106) 2024-09-13T11:40:21.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 1 -- 172.21.15.5:0/3716882545 learned_addr learned my addr 172.21.15.5:0/3716882545 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:21.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 10 monclient(hunting): get_auth_request con 0x7fe42c137930 auth_method 0 2024-09-13T11:40:21.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42a7fc640 10 monclient(hunting): get_auth_request con 0x7fe42c137d10 auth_method 0 2024-09-13T11:40:21.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42b7fe640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c13cd80 0x7fe42c138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42b7fe640 10 monclient(hunting): get_auth_request con 0x7fe42c13cd80 auth_method 0 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 10 monclient(hunting): handle_auth_done global_id 4482 payload 824 2024-09-13T11:40:21.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 1 -- 172.21.15.5:0/3716882545 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137d10 msgr2=0x7fe42c136920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137d10 0x7fe42c136920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 1 -- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c13cd80 msgr2=0x7fe42c138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c13cd80 0x7fe42c138d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 10 monclient: _finish_hunting 0 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 1 monclient: found mon.noname-a 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 1 -- 172.21.15.5:0/3716882545 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42c13b3d0 con 0x7fe42c137930 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42a7fc640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe42c137d10 0x7fe42c136920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe42affd640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe42c137930 0x7fe42c1cfc10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe41c01aaf0 tx=0x7fe41c0191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c0b0040 con 0x7fe42c137930 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:21.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 10 monclient: dump: 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 10 monclient: _finish_auth 0 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 5 monclient: authenticate success, global_id 4482 2024-09-13T11:40:21.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe41c0b9070 con 0x7fe42c137930 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient: _renew_subs 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 -- 172.21.15.5:0/3716882545 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe42c13c620 con 0x7fe42c137930 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c0aa990 con 0x7fe42c137930 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe423fff640 10 monclient: dump: 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.552 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe428ff9640 4 set_mon_vals no callback set 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient: _renew_subs 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 -- 172.21.15.5:0/3716882545 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe42c1402b0 con 0x7fe42c137930 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c136e60 0x7fe42c1392a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.531+0000 7fe4319cb640 1 -- 172.21.15.5:0/3716882545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c0a1360 con 0x7fe42c136e60 2024-09-13T11:40:21.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fe41c0aab30 con 0x7fe42c137930 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42a7fc640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c136e60 0x7fe42c1392a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42a7fc640 10 monclient: get_auth_request con 0x7fe42c136e60 auth_method 0 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 1 --2- 172.21.15.5:0/3716882545 >> v2:172.21.15.39:6812/1340838781 conn(0x7fe40c03b2a0 0x7fe40c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (secure 0 0 0) 0x7fe41c0e8e80 con 0x7fe42c137930 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42b7fe640 1 --2- 172.21.15.5:0/3716882545 >> v2:172.21.15.39:6812/1340838781 conn(0x7fe40c03b2a0 0x7fe40c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42b7fe640 10 monclient: get_auth_request con 0x7fe40c03b2a0 auth_method 0 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42a7fc640 10 monclient(hunting): handle_auth_done global_id 4393 payload 824 2024-09-13T11:40:21.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42a7fc640 1 -- 172.21.15.5:0/3716882545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe42c0a9210 con 0x7fe42c136e60 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42a7fc640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c136e60 0x7fe42c1392a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe42c13c830 tx=0x7fe41400ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe42b7fe640 1 --2- 172.21.15.5:0/3716882545 >> v2:172.21.15.39:6812/1340838781 conn(0x7fe40c03b2a0 0x7fe40c03d760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0a2990 tx=0x7fe424009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414018020 con 0x7fe42c136e60 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 10 monclient: dump: 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.555 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.556 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 <== mon.1 v2:172.21.15.39:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe41400da40 con 0x7fe42c136e60 2024-09-13T11:40:21.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:21.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:21.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c136e60 msgr2=0x7fe42c1392a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c136e60 0x7fe42c1392a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe42c13c830 tx=0x7fe41400ff70 comp rx=0 tx=0).stop 2024-09-13T11:40:21.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c136e60 msgr2=0x7fe42c1392a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:40:21.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.535+0000 7fe423fff640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c136e60 0x7fe42c1392a0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe42c13c830 tx=0x7fe41400ff70 comp rx=0 tx=0).stop 2024-09-13T11:40:21.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:40:21.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c1397e0 0x7fe42c0a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 1 -- 172.21.15.5:0/3716882545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c139bf0 con 0x7fe42c1397e0 2024-09-13T11:40:21.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42a7fc640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c1397e0 0x7fe42c0a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42a7fc640 10 monclient: get_auth_request con 0x7fe42c1397e0 auth_method 0 2024-09-13T11:40:21.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42a7fc640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42a7fc640 1 -- 172.21.15.5:0/3716882545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe42c09cc90 con 0x7fe42c1397e0 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42a7fc640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c1397e0 0x7fe42c0a2a50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe414019000 tx=0x7fe414003700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414018030 con 0x7fe42c1397e0 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe423fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe423fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe423fff640 10 monclient: dump: 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 <== mon.1 v2:172.21.15.39:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4866 (secure 0 0 0) 0x7fe414014a40 con 0x7fe42c1397e0 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe423fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe423fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c1397e0 msgr2=0x7fe42c0a2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe423fff640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c1397e0 0x7fe42c0a2a50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe414019000 tx=0x7fe414003700 comp rx=0 tx=0).stop 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe423fff640 1 -- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c1397e0 msgr2=0x7fe42c0a2a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:40:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe423fff640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe42c1397e0 0x7fe42c0a2a50 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe414019000 tx=0x7fe414003700 comp rx=0 tx=0).stop 2024-09-13T11:40:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:40:21.561 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-13T11:40:21.561 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-13T11:40:21.561 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-13T11:40:21.561 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 8, 2024-09-13T11:40:21.561 INFO:teuthology.orchestra.run.smithi005.stdout: "uptime": 33475, 2024-09-13T11:40:21.561 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-13T11:40:21.561 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-13T11:40:21.561 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-13T11:40:21.561 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-13T11:40:21.561 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 28, 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2450521708007866372", 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:21.562 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:21.563 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:21.564 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.564 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-13T11:40:21.564 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-13T11:40:21.564 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-13T11:40:21.564 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-13T11:40:21.564 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-13T11:40:21.564 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "59046471-79a9-4d90-9616-da0b446053a7", 2024-09-13T11:40:21.564 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:40:21.564 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 19, 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "squid", 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 1, 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:21.565 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-13T11:40:21.566 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.567 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:3300", 2024-09-13T11:40:21.568 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789", 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789/0", 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.39:6789/0", 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:21.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:21.570 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.571 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:21.572 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 7 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-13T11:40:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:40:21.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 1 -- 172.21.15.5:0/3716882545 >> v2:172.21.15.39:6812/1340838781 conn(0x7fe40c03b2a0 msgr2=0x7fe40c03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 1 --2- 172.21.15.5:0/3716882545 >> v2:172.21.15.39:6812/1340838781 conn(0x7fe40c03b2a0 0x7fe40c03d760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0a2990 tx=0x7fe424009040 comp rx=0 tx=0).stop 2024-09-13T11:40:21.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 10 monclient: shutdown 2024-09-13T11:40:21.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:21.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 1 -- 172.21.15.5:0/3716882545 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe42c137930 msgr2=0x7fe42c1cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 1 --2- 172.21.15.5:0/3716882545 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe42c137930 0x7fe42c1cfc10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe41c01aaf0 tx=0x7fe41c0191e0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42b7fe640 1 -- 172.21.15.5:0/3716882545 reap_dead start 2024-09-13T11:40:21.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe42b7fe640 1 -- 172.21.15.5:0/3716882545 reap_dead start 2024-09-13T11:40:21.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 1 -- 172.21.15.5:0/3716882545 shutdown_connections 2024-09-13T11:40:21.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 1 -- 172.21.15.5:0/3716882545 >> 172.21.15.5:0/3716882545 conn(0x7fe42c0a6c70 msgr2=0x7fe42c0a8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:21.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 1 -- 172.21.15.5:0/3716882545 shutdown_connections 2024-09-13T11:40:21.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.543+0000 7fe4319cb640 1 -- 172.21.15.5:0/3716882545 wait complete. 2024-09-13T11:40:21.576 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:40:21.576 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-13T11:40:21.576 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-13T11:40:21.576 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-13T11:40:21.577 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-13T11:40:21.577 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T11:40:21.577 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:40:21.603 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T11:40:21.603 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-13T11:40:21.603 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:40:21.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient: get_monmap_and_config 2024-09-13T11:40:21.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient: build_initial_monmap 2024-09-13T11:40:21.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient: monmap: 2024-09-13T11:40:21.724 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:21.724 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.724 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:21.726178+0000 2024-09-13T11:40:21.724 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:21.726178+0000 2024-09-13T11:40:21.724 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:21.724 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.725 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:21.725 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:21.725 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:21.725 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:21.725 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:21.725 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 1 Processor -- start 2024-09-13T11:40:21.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 1 -- start start 2024-09-13T11:40:21.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient: init 2024-09-13T11:40:21.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:21.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-13T11:40:21.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa168137930 0x7fa16809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient(hunting): picked mon.noname-c con 0x7fa168137930 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:21.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa168137d10 0x7fa1681380f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient(hunting): picked mon.noname-a con 0x7fa168137d10 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:21.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1680a4a70 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16809bef0 con 0x7fa1680a4a70 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa1680a5fe0 con 0x7fa168137d10 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fa16809c540 con 0x7fa168137930 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16fa1b640 10 monclient(hunting): authenticate will time out at 1025.074927s 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa168137d10 0x7fa1681380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16dfb8640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa1680a4a70 0x7fa1680a4ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:56430/0 (socket says 172.21.15.5:56430) 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16d7b7640 1 -- >> v2:172.21.15.39:3300/0 conn(0x7fa168137930 msgr2=0x7fa16809cc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16dfb8640 1 -- 172.21.15.5:0/879133444 learned_addr learned my addr 172.21.15.5:0/879133444 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 10 monclient(hunting): get_auth_request con 0x7fa168137d10 auth_method 0 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16d7b7640 1 --2- 172.21.15.5:0/879133444 >> v2:172.21.15.39:3300/0 conn(0x7fa168137930 0x7fa16809cc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 10 monclient(hunting): handle_auth_done global_id 4485 payload 274 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 1 -- 172.21.15.5:0/879133444 >> v2:172.21.15.39:3300/0 conn(0x7fa168137930 msgr2=0x7fa16809cc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 1 --2- 172.21.15.5:0/879133444 >> v2:172.21.15.39:3300/0 conn(0x7fa168137930 0x7fa16809cc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 1 -- 172.21.15.5:0/879133444 >> v1:172.21.15.5:6789/0 conn(0x7fa1680a4a70 legacy=0x7fa1680a4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 10 monclient: _finish_hunting 0 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 1 monclient: found mon.noname-a 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa15ffff640 1 -- 172.21.15.5:0/879133444 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4208168324 0 0) 0x7fa16809bef0 con 0x7fa1680a4a70 2024-09-13T11:40:21.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:21.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:21.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa16cfb6640 1 -- 172.21.15.5:0/879133444 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16809c1d0 con 0x7fa168137d10 2024-09-13T11:40:21.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.723+0000 7fa15ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:40:21.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16cfb6640 1 --2- 172.21.15.5:0/879133444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa168137d10 0x7fa1681380f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa1680a7cb0 tx=0x7fa15801c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e45b72c2ce2c7018 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 1 -- 172.21.15.5:0/879133444 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1580b0070 con 0x7fa168137d10 2024-09-13T11:40:21.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:21.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 10 monclient: dump: 2024-09-13T11:40:21.735 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.735 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.735 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.735 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.735 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.735 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.735 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 10 monclient: _finish_auth 0 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 1 -- 172.21.15.5:0/879133444 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1580ab600 con 0x7fa168137d10 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 5 monclient: authenticate success, global_id 4485 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 1 -- 172.21.15.5:0/879133444 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1580ab9d0 con 0x7fa168137d10 2024-09-13T11:40:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 4 set_mon_vals no callback set 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa15ffff640 10 monclient: dump: 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient: shutdown 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 -- 172.21.15.5:0/879133444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa168137d10 msgr2=0x7fa1681380f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 --2- 172.21.15.5:0/879133444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa168137d10 0x7fa1681380f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa1680a7cb0 tx=0x7fa15801c390 comp rx=0 tx=0).stop 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 -- 172.21.15.5:0/879133444 shutdown_connections 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 --2- 172.21.15.5:0/879133444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa168137d10 0x7fa1681380f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 --2- 172.21.15.5:0/879133444 >> v2:172.21.15.39:3300/0 conn(0x7fa168137930 0x7fa16809cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 -- 172.21.15.5:0/879133444 >> 172.21.15.5:0/879133444 conn(0x7fa1680a6c70 msgr2=0x7fa1680a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 -- 172.21.15.5:0/879133444 shutdown_connections 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 -- 172.21.15.5:0/879133444 wait complete. 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient: build_initial_monmap 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient: monmap: 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 Processor -- start 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 -- start start 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient: init 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:40:21.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1680a4a70 0x7fa168136100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient(hunting): picked mon.noname-c con 0x7fa1680a4a70 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa168137930 0x7fa168136640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient(hunting): picked mon.noname-a con 0x7fa168137930 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa168137d10 0x7fa16813c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient(hunting): picked mon.noname-b con 0x7fa168137d10 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa168142740 con 0x7fa168137930 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa1681425c0 con 0x7fa168137d10 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa1681428c0 con 0x7fa1680a4a70 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:21.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16fa1b640 10 monclient(hunting): authenticate will time out at 1025.079855s 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.727+0000 7fa16d7b7640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1680a4a70 0x7fa168136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1680a4a70 0x7fa168136100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47122/0 (socket says 172.21.15.5:47122) 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 1 -- 172.21.15.5:0/1034272879 learned_addr learned my addr 172.21.15.5:0/1034272879 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16dfb8640 1 -- 172.21.15.5:0/1034272879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa168137d10 msgr2=0x7fa16813c7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 10 monclient(hunting): get_auth_request con 0x7fa1680a4a70 auth_method 0 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16dfb8640 1 --2- 172.21.15.5:0/1034272879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa168137d10 0x7fa16813c7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16cfb6640 1 --2- 172.21.15.5:0/1034272879 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa168137930 0x7fa168136640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16cfb6640 10 monclient(hunting): get_auth_request con 0x7fa168137930 auth_method 0 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16cfb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16cfb6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16cfb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16cfb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16cfb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16cfb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 10 monclient(hunting): handle_auth_done global_id 4361 payload 824 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 1 -- 172.21.15.5:0/1034272879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa168137d10 msgr2=0x7fa16813c7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 1 --2- 172.21.15.5:0/1034272879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa168137d10 0x7fa16813c7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 1 -- 172.21.15.5:0/1034272879 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa168137930 msgr2=0x7fa168136640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 1 --2- 172.21.15.5:0/1034272879 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa168137930 0x7fa168136640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 10 monclient: _finish_hunting 0 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 1 monclient: found mon.noname-c 2024-09-13T11:40:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:21.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:21.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 1 -- 172.21.15.5:0/1034272879 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa168120aa0 con 0x7fa1680a4a70 2024-09-13T11:40:21.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16d7b7640 1 --2- 172.21.15.5:0/1034272879 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1680a4a70 0x7fa168136100 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa16400b560 tx=0x7fa16400ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 1 -- 172.21.15.5:0/1034272879 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164013020 con 0x7fa1680a4a70 2024-09-13T11:40:21.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:21.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 10 monclient: dump: 2024-09-13T11:40:21.743 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16fa1b640 5 monclient: authenticate success, global_id 4361 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 1 -- 172.21.15.5:0/1034272879 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa164003df0 con 0x7fa1680a4a70 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 1 -- 172.21.15.5:0/1034272879 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1640052b0 con 0x7fa1680a4a70 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15f7fe640 4 set_mon_vals no callback set 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 10 monclient: dump: 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16fa1b640 10 monclient: _renew_subs 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16fa1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16fa1b640 1 -- 172.21.15.5:0/1034272879 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa16806f930 con 0x7fa1680a4a70 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16fa1b640 10 monclient: _renew_subs 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16fa1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16fa1b640 1 -- 172.21.15.5:0/1034272879 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1680a1360 con 0x7fa1680a4a70 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16fa1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16fa1b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16fa1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16fa1b640 1 -- 172.21.15.5:0/1034272879 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa16809cc90 con 0x7fa1680a4a70 2024-09-13T11:40:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 1 -- 172.21.15.5:0/1034272879 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fa164016020 con 0x7fa1680a4a70 2024-09-13T11:40:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 1 --2- 172.21.15.5:0/1034272879 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa14c03b280 0x7fa14c03d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa15dffb640 1 -- 172.21.15.5:0/1034272879 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (secure 0 0 0) 0x7fa16404ead0 con 0x7fa1680a4a70 2024-09-13T11:40:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16cfb6640 1 --2- 172.21.15.5:0/1034272879 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa14c03b280 0x7fa14c03d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.731+0000 7fa16cfb6640 10 monclient: get_auth_request con 0x7fa14c03b280 auth_method 0 2024-09-13T11:40:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.735+0000 7fa16cfb6640 1 --2- 172.21.15.5:0/1034272879 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa14c03b280 0x7fa14c03d740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa1580ad000 tx=0x7fa1580a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.735+0000 7fa15dffb640 1 -- 172.21.15.5:0/1034272879 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fa16401d020 con 0x7fa1680a4a70 2024-09-13T11:40:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.735+0000 7fa15dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:21.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.735+0000 7fa15dffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:21.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.967+0000 7fa16fa1b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:40:21.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.967+0000 7fa16fa1b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:40:21.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.967+0000 7fa16fa1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:21.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.967+0000 7fa16fa1b640 1 -- 172.21.15.5:0/1034272879 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa16806cbe0 con 0x7fa1680a4a70 2024-09-13T11:40:21.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa15dffb640 1 -- 172.21.15.5:0/1034272879 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1548 (secure 0 0 0) 0x7fa164017e20 con 0x7fa1680a4a70 2024-09-13T11:40:21.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa15dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:40:21.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa15dffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:21.971 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","modified":"2024-09-13T11:39:20.750174Z","created":"2024-09-13T11:39:20.750174Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:40:21.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 1 -- 172.21.15.5:0/1034272879 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa14c03b280 msgr2=0x7fa14c03d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 1 --2- 172.21.15.5:0/1034272879 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa14c03b280 0x7fa14c03d740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa1580ad000 tx=0x7fa1580a6000 comp rx=0 tx=0).stop 2024-09-13T11:40:21.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 10 monclient: shutdown 2024-09-13T11:40:21.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:21.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 1 -- 172.21.15.5:0/1034272879 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1680a4a70 msgr2=0x7fa168136100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:21.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 1 --2- 172.21.15.5:0/1034272879 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1680a4a70 0x7fa168136100 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa16400b560 tx=0x7fa16400ba20 comp rx=0 tx=0).stop 2024-09-13T11:40:21.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 1 -- 172.21.15.5:0/1034272879 shutdown_connections 2024-09-13T11:40:21.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 1 --2- 172.21.15.5:0/1034272879 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa14c03b280 0x7fa14c03d740 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 1 --2- 172.21.15.5:0/1034272879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa168137d10 0x7fa16813c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 1 --2- 172.21.15.5:0/1034272879 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa168137930 0x7fa168136640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 1 --2- 172.21.15.5:0/1034272879 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa1680a4a70 0x7fa168136100 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:21.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 1 -- 172.21.15.5:0/1034272879 >> 172.21.15.5:0/1034272879 conn(0x7fa1680a6c70 msgr2=0x7fa1680a9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:21.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 1 -- 172.21.15.5:0/1034272879 shutdown_connections 2024-09-13T11:40:21.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:21.971+0000 7fa16fa1b640 1 -- 172.21.15.5:0/1034272879 wait complete. 2024-09-13T11:40:24.989 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:40:25.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient: get_monmap_and_config 2024-09-13T11:40:25.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient: build_initial_monmap 2024-09-13T11:40:25.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient: monmap: 2024-09-13T11:40:25.124 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:25.124 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:25.124 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:25.126157+0000 2024-09-13T11:40:25.124 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:25.126157+0000 2024-09-13T11:40:25.124 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:25.125 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:25.125 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:25.125 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:25.125 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:25.125 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:25.125 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:25.125 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:25.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 1 Processor -- start 2024-09-13T11:40:25.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 1 -- start start 2024-09-13T11:40:25.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient: init 2024-09-13T11:40:25.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:25.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-13T11:40:25.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf60134810 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:25.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faf60137660 0x7faf60139a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:25.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient(hunting): picked mon.noname-a con 0x7faf60137660 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faf60139f90 0x7faf6013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient(hunting): picked mon.noname-c con 0x7faf60139f90 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf6009ce60 con 0x7faf60134810 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7faf600a0950 con 0x7faf60137660 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7faf6009eeb0 con 0x7faf60139f90 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf64f78640 10 monclient(hunting): authenticate will time out at 1028.474846s 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faf60137660 0x7faf60139a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5ed9d640 1 -- >> v2:172.21.15.39:3300/0 conn(0x7faf60139f90 msgr2=0x7faf6013c380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5e59c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7faf60134810 0x7faf60134bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:56436/0 (socket says 172.21.15.5:56436) 2024-09-13T11:40:25.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5e59c640 1 -- 172.21.15.5:0/2126537026 learned_addr learned my addr 172.21.15.5:0/2126537026 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 10 monclient(hunting): get_auth_request con 0x7faf60137660 auth_method 0 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5ed9d640 1 --2- 172.21.15.5:0/2126537026 >> v2:172.21.15.39:3300/0 conn(0x7faf60139f90 0x7faf6013c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 10 monclient(hunting): handle_auth_done global_id 4494 payload 274 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 1 -- 172.21.15.5:0/2126537026 >> v2:172.21.15.39:3300/0 conn(0x7faf60139f90 msgr2=0x7faf6013c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 1 --2- 172.21.15.5:0/2126537026 >> v2:172.21.15.39:3300/0 conn(0x7faf60139f90 0x7faf6013c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 1 -- 172.21.15.5:0/2126537026 >> v1:172.21.15.5:6789/0 conn(0x7faf60134810 legacy=0x7faf60134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 10 monclient: _finish_hunting 0 2024-09-13T11:40:25.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 1 monclient: found mon.noname-a 2024-09-13T11:40:25.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:25.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5d59a640 1 -- 172.21.15.5:0/2126537026 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 377423879 0 0) 0x7faf6009ce60 con 0x7faf60134810 2024-09-13T11:40:25.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:25.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5dd9b640 1 -- 172.21.15.5:0/2126537026 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf6009e1e0 con 0x7faf60137660 2024-09-13T11:40:25.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.123+0000 7faf5d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:40:25.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5dd9b640 1 --2- 172.21.15.5:0/2126537026 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf60137660 0x7faf60139a50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faf6009e5c0 tx=0x7faf5001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b9cc16a124a7acc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:25.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 1 -- 172.21.15.5:0/2126537026 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf500b0070 con 0x7faf60137660 2024-09-13T11:40:25.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:25.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:25.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 10 monclient: dump: 2024-09-13T11:40:25.135 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:25.135 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:25.135 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:25.135 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 10 monclient: _finish_auth 0 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 10 monclient: _check_auth_tickets 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 1 -- 172.21.15.5:0/2126537026 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf500ab600 con 0x7faf60137660 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 5 monclient: authenticate success, global_id 4494 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 1 -- 172.21.15.5:0/2126537026 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf500ab9d0 con 0x7faf60137660 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 4 set_mon_vals no callback set 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf5d59a640 10 monclient: dump: 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient: shutdown 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 -- 172.21.15.5:0/2126537026 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf60137660 msgr2=0x7faf60139a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 --2- 172.21.15.5:0/2126537026 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf60137660 0x7faf60139a50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faf6009e5c0 tx=0x7faf5001c360 comp rx=0 tx=0).stop 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 -- 172.21.15.5:0/2126537026 shutdown_connections 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 --2- 172.21.15.5:0/2126537026 >> v2:172.21.15.39:3300/0 conn(0x7faf60139f90 0x7faf6013c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 --2- 172.21.15.5:0/2126537026 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf60137660 0x7faf60139a50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 -- 172.21.15.5:0/2126537026 >> 172.21.15.5:0/2126537026 conn(0x7faf600a2930 msgr2=0x7faf600a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 -- 172.21.15.5:0/2126537026 shutdown_connections 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 -- 172.21.15.5:0/2126537026 wait complete. 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient: build_initial_monmap 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient: monmap: 2024-09-13T11:40:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 Processor -- start 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 -- start start 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient: init 2024-09-13T11:40:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf60134810 0x7faf601cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient(hunting): picked mon.noname-c con 0x7faf60134810 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf60137660 0x7faf601d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient(hunting): picked mon.noname-b con 0x7faf60137660 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf60139f90 0x7faf6013d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient(hunting): picked mon.noname-a con 0x7faf60139f90 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7faf6013d460 con 0x7faf60139f90 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7faf6013d2e0 con 0x7faf60137660 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7faf6013d5e0 con 0x7faf60134810 2024-09-13T11:40:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.127+0000 7faf64f78640 10 monclient(hunting): authenticate will time out at 1028.479936s 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf60134810 0x7faf601cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5ed9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf60139f90 0x7faf6013d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5dd9b640 1 -- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf60137660 msgr2=0x7faf601d00a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5dd9b640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf60137660 0x7faf601d00a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf60134810 0x7faf601cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47138/0 (socket says 172.21.15.5:47138) 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 1 -- 172.21.15.5:0/288502689 learned_addr learned my addr 172.21.15.5:0/288502689 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 10 monclient(hunting): get_auth_request con 0x7faf60134810 auth_method 0 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5ed9d640 10 monclient(hunting): get_auth_request con 0x7faf60139f90 auth_method 0 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 10 monclient(hunting): handle_auth_done global_id 4364 payload 824 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 1 -- 172.21.15.5:0/288502689 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf60137660 msgr2=0x7faf601d00a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 1 --2- 172.21.15.5:0/288502689 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf60137660 0x7faf601d00a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 1 -- 172.21.15.5:0/288502689 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf60139f90 msgr2=0x7faf6013d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 1 --2- 172.21.15.5:0/288502689 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf60139f90 0x7faf6013d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 10 monclient: _finish_hunting 0 2024-09-13T11:40:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 1 monclient: found mon.noname-c 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 1 -- 172.21.15.5:0/288502689 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf60072530 con 0x7faf60134810 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5ed9d640 1 --2- 172.21.15.5:0/288502689 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf60139f90 0x7faf6013d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5e59c640 1 --2- 172.21.15.5:0/288502689 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf60134810 0x7faf601cfb60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf4800c490 tx=0x7faf4800c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 1 -- 172.21.15.5:0/288502689 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf48018070 con 0x7faf60134810 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 10 monclient: dump: 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 10 monclient: _finish_auth 0 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 1 -- 172.21.15.5:0/288502689 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf48007500 con 0x7faf60134810 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 1 -- 172.21.15.5:0/288502689 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf48004a30 con 0x7faf60134810 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 10 monclient: dump: 2024-09-13T11:40:25.144 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf64f78640 5 monclient: authenticate success, global_id 4364 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5cd99640 4 set_mon_vals no callback set 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf64f78640 10 monclient: _renew_subs 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf64f78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf64f78640 1 -- 172.21.15.5:0/288502689 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf6006dbf0 con 0x7faf60134810 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf64f78640 10 monclient: _renew_subs 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf64f78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf64f78640 1 -- 172.21.15.5:0/288502689 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faf60072250 con 0x7faf60134810 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 1 -- 172.21.15.5:0/288502689 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7faf48004bd0 con 0x7faf60134810 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf64f78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf64f78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf64f78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf64f78640 1 -- 172.21.15.5:0/288502689 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf6009dfa0 con 0x7faf60134810 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 1 --2- 172.21.15.5:0/288502689 >> v2:172.21.15.39:6812/1340838781 conn(0x7faf4003b250 0x7faf4003d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf56ffd640 1 -- 172.21.15.5:0/288502689 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (secure 0 0 0) 0x7faf4804dfe0 con 0x7faf60134810 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5dd9b640 1 --2- 172.21.15.5:0/288502689 >> v2:172.21.15.39:6812/1340838781 conn(0x7faf4003b250 0x7faf4003d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5dd9b640 10 monclient: get_auth_request con 0x7faf4003b250 auth_method 0 2024-09-13T11:40:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.131+0000 7faf5dd9b640 1 --2- 172.21.15.5:0/288502689 >> v2:172.21.15.39:6812/1340838781 conn(0x7faf4003b250 0x7faf4003d710 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faf500ad000 tx=0x7faf50008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:25.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.135+0000 7faf56ffd640 1 -- 172.21.15.5:0/288502689 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7faf4801bc60 con 0x7faf60134810 2024-09-13T11:40:25.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.135+0000 7faf56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:25.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.135+0000 7faf56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:25.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:40:25.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:40:25.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:25.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 -- 172.21.15.5:0/288502689 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faf60134bf0 con 0x7faf60134810 2024-09-13T11:40:25.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf56ffd640 1 -- 172.21.15.5:0/288502689 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1548 (secure 0 0 0) 0x7faf4801b740 con 0x7faf60134810 2024-09-13T11:40:25.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:40:25.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf56ffd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:25.376 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":32,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","modified":"2024-09-13T11:39:20.750174Z","created":"2024-09-13T11:39:20.750174Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:40:25.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 -- 172.21.15.5:0/288502689 >> v2:172.21.15.39:6812/1340838781 conn(0x7faf4003b250 msgr2=0x7faf4003d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:25.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 --2- 172.21.15.5:0/288502689 >> v2:172.21.15.39:6812/1340838781 conn(0x7faf4003b250 0x7faf4003d710 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faf500ad000 tx=0x7faf50008040 comp rx=0 tx=0).stop 2024-09-13T11:40:25.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 10 monclient: shutdown 2024-09-13T11:40:25.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:25.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 -- 172.21.15.5:0/288502689 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf60134810 msgr2=0x7faf601cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:25.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 --2- 172.21.15.5:0/288502689 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf60134810 0x7faf601cfb60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf4800c490 tx=0x7faf4800c950 comp rx=0 tx=0).stop 2024-09-13T11:40:25.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 -- 172.21.15.5:0/288502689 shutdown_connections 2024-09-13T11:40:25.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 --2- 172.21.15.5:0/288502689 >> v2:172.21.15.39:6812/1340838781 conn(0x7faf4003b250 0x7faf4003d710 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:25.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 --2- 172.21.15.5:0/288502689 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf60139f90 0x7faf6013d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:25.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 --2- 172.21.15.5:0/288502689 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf60137660 0x7faf601d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:25.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 --2- 172.21.15.5:0/288502689 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf60134810 0x7faf601cfb60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:25.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 -- 172.21.15.5:0/288502689 >> 172.21.15.5:0/288502689 conn(0x7faf600a2930 msgr2=0x7faf600a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:25.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 -- 172.21.15.5:0/288502689 shutdown_connections 2024-09-13T11:40:25.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:25.375+0000 7faf64f78640 1 -- 172.21.15.5:0/288502689 wait complete. 2024-09-13T11:40:28.396 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:40:28.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.527+0000 7f25e640b640 10 monclient: get_monmap_and_config 2024-09-13T11:40:28.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.527+0000 7f25e640b640 10 monclient: build_initial_monmap 2024-09-13T11:40:28.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.527+0000 7f25e640b640 10 monclient: monmap: 2024-09-13T11:40:28.529 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:28.529 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:28.530 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:28.531371+0000 2024-09-13T11:40:28.530 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:28.531371+0000 2024-09-13T11:40:28.530 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:28.530 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:28.530 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:28.530 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:28.530 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:28.530 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:28.530 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:28.531 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:28.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.527+0000 7f25e640b640 1 Processor -- start 2024-09-13T11:40:28.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.527+0000 7f25e640b640 1 -- start start 2024-09-13T11:40:28.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.527+0000 7f25e640b640 10 monclient: init 2024-09-13T11:40:28.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:28.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-13T11:40:28.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25e0134810 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:28.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f25e0137660 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:28.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25e013b170 0x7f25e013d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:28.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 10 monclient(hunting): picked mon.noname-b con 0x7f25e013b170 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:28.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25e009ce60 con 0x7f25e0137660 2024-09-13T11:40:28.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25e009eeb0 con 0x7f25e0134810 2024-09-13T11:40:28.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:28.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f25e00a0950 con 0x7f25e013b170 2024-09-13T11:40:28.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:28.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 10 monclient(hunting): authenticate will time out at 1031.880349s 2024-09-13T11:40:28.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25dffff640 1 -- >> v1:172.21.15.39:6789/0 conn(0x7f25e0134810 legacy=0x7f25e0134bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.39:6789/0 2024-09-13T11:40:28.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25df7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f25e0137660 0x7f25e0139a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:44686/0 (socket says 172.21.15.5:44686) 2024-09-13T11:40:28.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25df7fe640 1 -- 172.21.15.5:0/2080243656 learned_addr learned my addr 172.21.15.5:0/2080243656 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:28.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 1 --2- 172.21.15.5:0/2080243656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25e013b170 0x7f25e013d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:28.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 10 monclient(hunting): get_auth_request con 0x7f25e013b170 auth_method 0 2024-09-13T11:40:28.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:28.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:28.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:28.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:28.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:28.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:28.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 10 monclient(hunting): handle_auth_done global_id 4367 payload 274 2024-09-13T11:40:28.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 1 -- 172.21.15.5:0/2080243656 >> v1:172.21.15.39:6789/0 conn(0x7f25e0134810 legacy=0x7f25e0134bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:28.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 1 -- 172.21.15.5:0/2080243656 >> v1:172.21.15.5:6789/0 conn(0x7f25e0137660 legacy=0x7f25e0139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:28.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 10 monclient: _finish_hunting 0 2024-09-13T11:40:28.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 1 monclient: found mon.noname-b 2024-09-13T11:40:28.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:28.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:40:28.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 1 -- 172.21.15.5:0/2080243656 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e009e1e0 con 0x7f25e013b170 2024-09-13T11:40:28.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e49a8640 1 --2- 172.21.15.5:0/2080243656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25e013b170 0x7f25e013d560 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f25d80097f0 tx=0x7f25d801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edbc69f9a2b5149d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:28.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 1 -- 172.21.15.5:0/2080243656 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d80b0070 con 0x7f25e013b170 2024-09-13T11:40:28.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:28.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:40:28.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 10 monclient: dump: 2024-09-13T11:40:28.538 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:28.538 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:28.538 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:28.538 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:28.538 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:28.538 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 10 monclient: _finish_auth 0 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 10 monclient: _check_auth_tickets 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 1 -- 172.21.15.5:0/2080243656 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25d80ab600 con 0x7f25e013b170 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 5 monclient: authenticate success, global_id 4367 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 4 set_mon_vals no callback set 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 1 -- 172.21.15.5:0/2080243656 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d80ab9d0 con 0x7f25e013b170 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25deffd640 10 monclient: dump: 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:28.540 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:28.541 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:28.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 10 monclient: shutdown 2024-09-13T11:40:28.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:28.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 1 -- 172.21.15.5:0/2080243656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25e013b170 msgr2=0x7f25e013d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:28.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 1 --2- 172.21.15.5:0/2080243656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25e013b170 0x7f25e013d560 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f25d80097f0 tx=0x7f25d801c3a0 comp rx=0 tx=0).stop 2024-09-13T11:40:28.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 1 -- 172.21.15.5:0/2080243656 shutdown_connections 2024-09-13T11:40:28.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 1 --2- 172.21.15.5:0/2080243656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25e013b170 0x7f25e013d560 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:28.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 1 -- 172.21.15.5:0/2080243656 >> 172.21.15.5:0/2080243656 conn(0x7f25e00a2930 msgr2=0x7f25e00a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:28.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 1 -- 172.21.15.5:0/2080243656 shutdown_connections 2024-09-13T11:40:28.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.531+0000 7f25e640b640 1 -- 172.21.15.5:0/2080243656 wait complete. 2024-09-13T11:40:28.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient: build_initial_monmap 2024-09-13T11:40:28.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient: monmap: 2024-09-13T11:40:28.542 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:28.542 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:28.542 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:28.542 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:28.542 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:28.542 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:28.542 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:28.542 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:28.542 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:28.542 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:28.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 1 Processor -- start 2024-09-13T11:40:28.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 1 -- start start 2024-09-13T11:40:28.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient: init 2024-09-13T11:40:28.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:28.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:28.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25e0134810 0x7f25e01335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:28.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient(hunting): picked mon.noname-c con 0x7f25e0134810 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:28.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e0137660 0x7f25e011c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:28.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient(hunting): picked mon.noname-b con 0x7f25e0137660 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:28.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25e013b170 0x7f25e011c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:28.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient(hunting): picked mon.noname-a con 0x7f25e013b170 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:28.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:28.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f25e01408e0 con 0x7f25e013b170 2024-09-13T11:40:28.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:28.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f25e00a0d30 con 0x7f25e0137660 2024-09-13T11:40:28.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:28.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f25e0140a60 con 0x7f25e0134810 2024-09-13T11:40:28.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:28.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient(hunting): authenticate will time out at 1031.885458s 2024-09-13T11:40:28.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25e013b170 0x7f25e011c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:28.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25df7fe640 1 -- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e0137660 msgr2=0x7f25e011c2b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:28.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dffff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25e0134810 0x7f25e01335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:28.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25df7fe640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e0137660 0x7f25e011c2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:28.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25e013b170 0x7f25e011c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49668/0 (socket says 172.21.15.5:49668) 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 1 -- 172.21.15.5:0/816045174 learned_addr learned my addr 172.21.15.5:0/816045174 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 10 monclient(hunting): get_auth_request con 0x7f25e013b170 auth_method 0 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dffff640 10 monclient(hunting): get_auth_request con 0x7f25e0134810 auth_method 0 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:28.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:28.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:28.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:28.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 10 monclient(hunting): handle_auth_done global_id 4503 payload 824 2024-09-13T11:40:28.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 1 -- 172.21.15.5:0/816045174 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25e0134810 msgr2=0x7f25e01335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:28.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 1 --2- 172.21.15.5:0/816045174 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25e0134810 0x7f25e01335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:28.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 1 -- 172.21.15.5:0/816045174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e0137660 msgr2=0x7f25e011c2b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:40:28.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 1 --2- 172.21.15.5:0/816045174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e0137660 0x7f25e011c2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:28.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 10 monclient: _finish_hunting 0 2024-09-13T11:40:28.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 1 monclient: found mon.noname-a 2024-09-13T11:40:28.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:28.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:28.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 1 -- 172.21.15.5:0/816045174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e011f730 con 0x7f25e013b170 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e49a8640 1 --2- 172.21.15.5:0/816045174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25e013b170 0x7f25e011c7f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f25d80097f0 tx=0x7f25d8019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 1 -- 172.21.15.5:0/816045174 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d80ba020 con 0x7f25e013b170 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 10 monclient: dump: 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:28.550 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 10 monclient: _finish_auth 0 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 10 monclient: _check_auth_tickets 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 5 monclient: authenticate success, global_id 4503 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 1 -- 172.21.15.5:0/816045174 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25d80b4400 con 0x7f25e013b170 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient: _renew_subs 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 1 -- 172.21.15.5:0/816045174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e011de30 con 0x7f25e013b170 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 1 -- 172.21.15.5:0/816045174 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d80b0040 con 0x7f25e013b170 2024-09-13T11:40:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:28.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:28.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 10 monclient: dump: 2024-09-13T11:40:28.552 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:28.552 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:28.552 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:28.552 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:28.552 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:28.552 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:28.552 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:28.552 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:28.552 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:28.552 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient: _renew_subs 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25de7fc640 4 set_mon_vals no callback set 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 1 -- 172.21.15.5:0/816045174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e006dbf0 con 0x7f25e013b170 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25e640b640 1 -- 172.21.15.5:0/816045174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e013aaa0 con 0x7f25e013b170 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.535+0000 7f25dcff9640 1 -- 172.21.15.5:0/816045174 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f25d80bd020 con 0x7f25e013b170 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.539+0000 7f25dcff9640 1 --2- 172.21.15.5:0/816045174 >> v2:172.21.15.39:6812/1340838781 conn(0x7f25c003b250 0x7f25c003d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.539+0000 7f25dcff9640 1 -- 172.21.15.5:0/816045174 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (secure 0 0 0) 0x7f25d80e92e0 con 0x7f25e013b170 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.539+0000 7f25dffff640 1 --2- 172.21.15.5:0/816045174 >> v2:172.21.15.39:6812/1340838781 conn(0x7f25c003b250 0x7f25c003d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:28.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.539+0000 7f25dffff640 10 monclient: get_auth_request con 0x7f25c003b250 auth_method 0 2024-09-13T11:40:28.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.539+0000 7f25dffff640 1 --2- 172.21.15.5:0/816045174 >> v2:172.21.15.39:6812/1340838781 conn(0x7f25c003b250 0x7f25c003d710 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009420 tx=0x7f25d000d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:28.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.543+0000 7f25dcff9640 1 -- 172.21.15.5:0/816045174 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f25e013aaa0 con 0x7f25e013b170 2024-09-13T11:40:28.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.543+0000 7f25dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:28.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.543+0000 7f25dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:28.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.775+0000 7f25e640b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:40:28.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.775+0000 7f25e640b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:40:28.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.775+0000 7f25e640b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:28.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.775+0000 7f25e640b640 1 -- 172.21.15.5:0/816045174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f25e0134bf0 con 0x7f25e013b170 2024-09-13T11:40:28.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.775+0000 7f25dcff9640 1 -- 172.21.15.5:0/816045174 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1548 (secure 0 0 0) 0x7f25d80c4030 con 0x7f25e013b170 2024-09-13T11:40:28.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.775+0000 7f25dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:40:28.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.775+0000 7f25dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:28.778 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":35,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","modified":"2024-09-13T11:39:20.750174Z","created":"2024-09-13T11:39:20.750174Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:40:28.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 1 -- 172.21.15.5:0/816045174 >> v2:172.21.15.39:6812/1340838781 conn(0x7f25c003b250 msgr2=0x7f25c003d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:28.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 1 --2- 172.21.15.5:0/816045174 >> v2:172.21.15.39:6812/1340838781 conn(0x7f25c003b250 0x7f25c003d710 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009420 tx=0x7f25d000d040 comp rx=0 tx=0).stop 2024-09-13T11:40:28.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 10 monclient: shutdown 2024-09-13T11:40:28.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:28.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 1 -- 172.21.15.5:0/816045174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25e013b170 msgr2=0x7f25e011c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:28.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 1 --2- 172.21.15.5:0/816045174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25e013b170 0x7f25e011c7f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f25d80097f0 tx=0x7f25d8019140 comp rx=0 tx=0).stop 2024-09-13T11:40:28.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 1 -- 172.21.15.5:0/816045174 shutdown_connections 2024-09-13T11:40:28.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 1 --2- 172.21.15.5:0/816045174 >> v2:172.21.15.39:6812/1340838781 conn(0x7f25c003b250 0x7f25c003d710 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:28.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 1 --2- 172.21.15.5:0/816045174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25e013b170 0x7f25e011c7f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:28.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 1 --2- 172.21.15.5:0/816045174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e0137660 0x7f25e011c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:28.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 1 --2- 172.21.15.5:0/816045174 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25e0134810 0x7f25e01335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:28.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 1 -- 172.21.15.5:0/816045174 >> 172.21.15.5:0/816045174 conn(0x7f25e00a2930 msgr2=0x7f25e00a4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:28.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 1 -- 172.21.15.5:0/816045174 shutdown_connections 2024-09-13T11:40:28.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:28.779+0000 7f25e640b640 1 -- 172.21.15.5:0/816045174 wait complete. 2024-09-13T11:40:31.796 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:40:31.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient: get_monmap_and_config 2024-09-13T11:40:31.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient: build_initial_monmap 2024-09-13T11:40:31.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient: monmap: 2024-09-13T11:40:31.928 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:31.928 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:31.929 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:31.930540+0000 2024-09-13T11:40:31.929 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:31.930540+0000 2024-09-13T11:40:31.929 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:31.929 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:31.929 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:31.929 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:31.929 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:31.929 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:31.929 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:31.929 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:31.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 1 Processor -- start 2024-09-13T11:40:31.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 1 -- start start 2024-09-13T11:40:31.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient: init 2024-09-13T11:40:31.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:31.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-13T11:40:31.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:31.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient(hunting): picked mon.noname-a con 0x7fd2240a0ea0 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:31.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:31.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient(hunting): picked mon.noname-b con 0x7fd2240a1ae0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:31.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2241359f0 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:31.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22406f010 con 0x7fd2241359f0 2024-09-13T11:40:31.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:31.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd22406f640 con 0x7fd2240a0ea0 2024-09-13T11:40:31.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:31.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fd2240a0000 con 0x7fd2240a1ae0 2024-09-13T11:40:31.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:31.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22b3d0640 10 monclient(hunting): authenticate will time out at 1035.279249s 2024-09-13T11:40:31.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22916c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:31.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22996d640 1 -- >> v1:172.21.15.39:6789/0 conn(0x7fd2241359f0 legacy=0x7fd22413c800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.39:6789/0 2024-09-13T11:40:31.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22916c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49678/0 (socket says 172.21.15.5:49678) 2024-09-13T11:40:31.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22916c640 1 -- 172.21.15.5:0/939506979 learned_addr learned my addr 172.21.15.5:0/939506979 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:31.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22916c640 10 monclient(hunting): get_auth_request con 0x7fd2240a0ea0 auth_method 0 2024-09-13T11:40:31.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22916c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22916c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22916c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22896b640 10 monclient(hunting): get_auth_request con 0x7fd2240a1ae0 auth_method 0 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22896b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22896b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22896b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22916c640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 msgr2=0x7fd2240a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22916c640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 msgr2=0x7fd2240a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22896b640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 msgr2=0x7fd2241354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22896b640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 msgr2=0x7fd2241354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:40:31.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:40:31.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:31.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:31.927+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:32.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22996d640 1 -- 172.21.15.5:0/939506979 >> v1:172.21.15.39:6789/0 conn(0x7fd2241359f0 legacy=0x7fd22413c800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.39:6789/0 2024-09-13T11:40:32.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:32.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:32.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22896b640 10 monclient(hunting): get_auth_request con 0x7fd2240a1ae0 auth_method 0 2024-09-13T11:40:32.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22896b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:32.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22896b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:32.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22896b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:32.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22916c640 10 monclient(hunting): get_auth_request con 0x7fd2240a0ea0 auth_method 0 2024-09-13T11:40:32.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22916c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:32.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22916c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:32.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22916c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:32.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22896b640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 msgr2=0x7fd2241354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:40:32.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22896b640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 msgr2=0x7fd2241354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:40:32.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:40:32.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22916c640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 msgr2=0x7fd2240a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:40:32.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22916c640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 msgr2=0x7fd2240a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:40:32.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:40:32.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:40:32.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.131+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:40:32.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:32.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:32.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22916c640 10 monclient(hunting): get_auth_request con 0x7fd2240a0ea0 auth_method 0 2024-09-13T11:40:32.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22916c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:32.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22916c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:32.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22916c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22896b640 10 monclient(hunting): get_auth_request con 0x7fd2240a1ae0 auth_method 0 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22896b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22896b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22896b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22916c640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 msgr2=0x7fd2240a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22916c640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 msgr2=0x7fd2240a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22896b640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 msgr2=0x7fd2241354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22896b640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 msgr2=0x7fd2241354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22996d640 1 -- 172.21.15.5:0/939506979 >> v1:172.21.15.39:6789/0 conn(0x7fd2241359f0 legacy=0x7fd22413c800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.39:6789/0 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:40:32.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:32.535+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:40:33.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22996d640 1 -- 172.21.15.5:0/939506979 >> v1:172.21.15.39:6789/0 conn(0x7fd2241359f0 legacy=0x7fd22413c800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.39:6789/0 2024-09-13T11:40:33.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:33.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:33.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22896b640 10 monclient(hunting): get_auth_request con 0x7fd2240a1ae0 auth_method 0 2024-09-13T11:40:33.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22896b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:33.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22896b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:33.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22896b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:33.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22916c640 10 monclient(hunting): get_auth_request con 0x7fd2240a0ea0 auth_method 0 2024-09-13T11:40:33.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22916c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:33.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22916c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:33.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22916c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:33.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22896b640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 msgr2=0x7fd2241354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:40:33.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22916c640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 msgr2=0x7fd2240a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T11:40:33.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22896b640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 msgr2=0x7fd2241354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:40:33.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22916c640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 msgr2=0x7fd2240a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:40:33.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:40:33.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:40:33.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:40:33.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:33.339+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T11:40:34.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 10 monclient(hunting): tick 2024-09-13T11:40:34.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-13T11:40:34.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 1 monclient(hunting): continuing hunt 2024-09-13T11:40:34.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-13T11:40:34.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 msgr2=0x7fd2241354b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:34.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2241354b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:34.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 msgr2=0x7fd2240a1280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:34.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.5:3300/0 conn(0x7fd2240a0ea0 0x7fd2240a1280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:34.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 1 -- 172.21.15.5:0/939506979 >> v1:172.21.15.39:6789/0 conn(0x7fd2241359f0 legacy=0x7fd22413c800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:34.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 10 monclient: _add_conns ranks=[0,4] 2024-09-13T11:40:34.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2080036b0 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:34.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.39:3300/0 conn(0x7fd208007400 0x7fd2080098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:34.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fd208007400 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:34.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 1 -- 172.21.15.5:0/939506979 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22409fd80 con 0x7fd2080036b0 2024-09-13T11:40:34.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:34.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 1 -- 172.21.15.5:0/939506979 --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fd22406f640 con 0x7fd208007400 2024-09-13T11:40:34.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd216ffd640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:34.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd22896b640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.39:3300/0 conn(0x7fd208007400 msgr2=0x7fd2080098a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:34.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:34.927+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.39:3300/0 conn(0x7fd208007400 0x7fd2080098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:35.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:35.131+0000 7fd22896b640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.39:3300/0 conn(0x7fd208007400 msgr2=0x7fd2080098a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:35.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:35.131+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.39:3300/0 conn(0x7fd208007400 0x7fd2080098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T11:40:35.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:35.535+0000 7fd22896b640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.39:3300/0 conn(0x7fd208007400 msgr2=0x7fd2080098a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:35.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:35.535+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.39:3300/0 conn(0x7fd208007400 0x7fd2080098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T11:40:36.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.219+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3836051468 0 0) 0x7fd22409fd80 con 0x7fd2080036b0 2024-09-13T11:40:36.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.219+0000 7fd217fff640 10 monclient(hunting): _negotiate 2024-09-13T11:40:36.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.219+0000 7fd217fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.219+0000 7fd217fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.219+0000 7fd217fff640 10 monclient(hunting): my global_id is 4506 2024-09-13T11:40:36.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.219+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd210003530 con 0x7fd2080036b0 2024-09-13T11:40:36.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 101199289 0 0) 0x7fd210003530 con 0x7fd2080036b0 2024-09-13T11:40:36.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 >> v2:172.21.15.39:3300/0 conn(0x7fd208007400 msgr2=0x7fd2080098a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:36.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 --2- 172.21.15.5:0/939506979 >> v2:172.21.15.39:3300/0 conn(0x7fd208007400 0x7fd2080098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:36.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:40:36.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:36.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-13T11:40:36.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd22406f010 con 0x7fd2080036b0 2024-09-13T11:40:36.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient: _finish_auth 0 2024-09-13T11:40:36.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient: _check_auth_tickets 2024-09-13T11:40:36.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:36.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3199362888 0 0) 0x7fd22000f040 con 0x7fd2080036b0 2024-09-13T11:40:36.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:40:36.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient: dump: 2024-09-13T11:40:36.226 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.226 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.226 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.226 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.226 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.226 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.226 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.227 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.227 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.227 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-13T11:40:36.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:36.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 >> v1:172.21.15.5:6789/0 conn(0x7fd2080036b0 legacy=0x7fd208005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:36.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2100039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22996d640 1 -- 172.21.15.5:0/939506979 reap_dead start 2024-09-13T11:40:36.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient(hunting): picked mon.c con 0x7fd2240a1ae0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:36.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2100040b0 0x7fd210004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient(hunting): picked mon.b con 0x7fd2100040b0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd210004e20 0x7fd2100052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient(hunting): picked mon.a con 0x7fd210004e20 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd22406f640 con 0x7fd210004e20 2024-09-13T11:40:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd21000bd30 con 0x7fd2100040b0 2024-09-13T11:40:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fd21000bf30 con 0x7fd2240a1ae0 2024-09-13T11:40:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd217fff640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:40:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2100039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 10 monclient(hunting): get_auth_request con 0x7fd2240a1ae0 auth_method 0 2024-09-13T11:40:36.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:36.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22896b640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2100040b0 msgr2=0x7fd210004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:36.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22896b640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2100040b0 0x7fd210004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:36.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:40:36.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22996d640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd210004e20 0x7fd2100052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22996d640 10 monclient(hunting): get_auth_request con 0x7fd210004e20 auth_method 0 2024-09-13T11:40:36.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22996d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22996d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22996d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:36.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 10 monclient(hunting): handle_auth_done global_id 4506 payload 293 2024-09-13T11:40:36.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2100040b0 msgr2=0x7fd210004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:36.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2100040b0 0x7fd210004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd210004e20 msgr2=0x7fd2100052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:36.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd210004e20 0x7fd2100052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 10 monclient: _finish_hunting 0 2024-09-13T11:40:36.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 1 monclient: found mon.c 2024-09-13T11:40:36.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:36.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:36.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 1 -- 172.21.15.5:0/939506979 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd210003530 con 0x7fd2240a1ae0 2024-09-13T11:40:36.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 10 monclient: _finish_auth 0 2024-09-13T11:40:36.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 10 monclient: _check_auth_tickets 2024-09-13T11:40:36.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22916c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:36.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22b3d0640 5 monclient: authenticate success, global_id 4506 2024-09-13T11:40:36.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22b3d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:36.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.223+0000 7fd22996d640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd210004e20 0x7fd2100052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:40:36.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.227+0000 7fd22916c640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2100039c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd220009100 tx=0x7fd22001fe80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a2a493f93b7041f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2200af070 con 0x7fd2240a1ae0 2024-09-13T11:40:36.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd217fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:36.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd217fff640 10 monclient: dump: 2024-09-13T11:40:36.233 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.233 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.233 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.233 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.234 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.234 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.234 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.234 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.234 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.234 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:36.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2200a9af0 con 0x7fd2240a1ae0 2024-09-13T11:40:36.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd217fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 4 set_mon_vals no callback set 2024-09-13T11:40:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd217fff640 1 -- 172.21.15.5:0/939506979 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22000cab0 con 0x7fd2240a1ae0 2024-09-13T11:40:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd217fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd217fff640 10 monclient: dump: 2024-09-13T11:40:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.236 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient: shutdown 2024-09-13T11:40:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 -- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 msgr2=0x7fd2100039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2100039c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd220009100 tx=0x7fd22001fe80 comp rx=0 tx=0).stop 2024-09-13T11:40:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 -- 172.21.15.5:0/939506979 shutdown_connections 2024-09-13T11:40:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd210004e20 0x7fd2100052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2100040b0 0x7fd210004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 --2- 172.21.15.5:0/939506979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd2240a1ae0 0x7fd2100039c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 -- 172.21.15.5:0/939506979 >> 172.21.15.5:0/939506979 conn(0x7fd2240a2930 msgr2=0x7fd2240a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 -- 172.21.15.5:0/939506979 shutdown_connections 2024-09-13T11:40:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 -- 172.21.15.5:0/939506979 wait complete. 2024-09-13T11:40:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient: build_initial_monmap 2024-09-13T11:40:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient: monmap: 2024-09-13T11:40:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.238 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:36.238 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:36.238 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:36.238 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 Processor -- start 2024-09-13T11:40:36.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 -- start start 2024-09-13T11:40:36.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient: init 2024-09-13T11:40:36.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:36.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:40:36.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2100040b0 0x7fd2241cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient(hunting): picked mon.noname-a con 0x7fd2100040b0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:36.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd210004e20 0x7fd2241ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient(hunting): picked mon.noname-c con 0x7fd210004e20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:36.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2240a1ae0 0x7fd224123250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient(hunting): picked mon.noname-b con 0x7fd2240a1ae0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:36.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd2240a0b20 con 0x7fd2100040b0 2024-09-13T11:40:36.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd2240a24b0 con 0x7fd2240a1ae0 2024-09-13T11:40:36.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fd2240a0ca0 con 0x7fd210004e20 2024-09-13T11:40:36.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:36.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.231+0000 7fd22b3d0640 10 monclient(hunting): authenticate will time out at 1039.583884s 2024-09-13T11:40:36.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22896b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd210004e20 0x7fd2241ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2100040b0 0x7fd2241cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22996d640 1 -- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2240a1ae0 msgr2=0x7fd224123250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:36.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2100040b0 0x7fd2241cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49716/0 (socket says 172.21.15.5:49716) 2024-09-13T11:40:36.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 1 -- 172.21.15.5:0/2292324328 learned_addr learned my addr 172.21.15.5:0/2292324328 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:36.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22996d640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2240a1ae0 0x7fd224123250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:36.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 10 monclient(hunting): get_auth_request con 0x7fd2100040b0 auth_method 0 2024-09-13T11:40:36.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22896b640 10 monclient(hunting): get_auth_request con 0x7fd210004e20 auth_method 0 2024-09-13T11:40:36.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22896b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22896b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22896b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:36.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22896b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22896b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22896b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:36.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 10 monclient(hunting): handle_auth_done global_id 4509 payload 824 2024-09-13T11:40:36.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 1 -- 172.21.15.5:0/2292324328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd210004e20 msgr2=0x7fd2241ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 1 --2- 172.21.15.5:0/2292324328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd210004e20 0x7fd2241ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 1 -- 172.21.15.5:0/2292324328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2240a1ae0 msgr2=0x7fd224123250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:40:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 1 --2- 172.21.15.5:0/2292324328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2240a1ae0 0x7fd224123250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 10 monclient: _finish_hunting 0 2024-09-13T11:40:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 1 monclient: found mon.noname-a 2024-09-13T11:40:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 1 -- 172.21.15.5:0/2292324328 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd224124d10 con 0x7fd2100040b0 2024-09-13T11:40:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22916c640 1 --2- 172.21.15.5:0/2292324328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2100040b0 0x7fd2241cc620 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd220009100 tx=0x7fd210005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 1 -- 172.21.15.5:0/2292324328 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2200c2030 con 0x7fd2100040b0 2024-09-13T11:40:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 10 monclient: dump: 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 1 -- 172.21.15.5:0/2292324328 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2200a98f0 con 0x7fd2100040b0 2024-09-13T11:40:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 1 -- 172.21.15.5:0/2292324328 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2200af040 con 0x7fd2100040b0 2024-09-13T11:40:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 10 monclient: dump: 2024-09-13T11:40:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.249 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd2177fe640 4 set_mon_vals no callback set 2024-09-13T11:40:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22b3d0640 5 monclient: authenticate success, global_id 4509 2024-09-13T11:40:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22b3d0640 10 monclient: _renew_subs 2024-09-13T11:40:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22b3d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22b3d0640 1 -- 172.21.15.5:0/2292324328 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2241267d0 con 0x7fd2100040b0 2024-09-13T11:40:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22b3d0640 10 monclient: _renew_subs 2024-09-13T11:40:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22b3d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22b3d0640 1 -- 172.21.15.5:0/2292324328 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd224125870 con 0x7fd2100040b0 2024-09-13T11:40:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22b3d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22b3d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22b3d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 1 -- 172.21.15.5:0/2292324328 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fd2200c5070 con 0x7fd2100040b0 2024-09-13T11:40:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22b3d0640 1 -- 172.21.15.5:0/2292324328 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2241355d0 con 0x7fd2100040b0 2024-09-13T11:40:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 1 --2- 172.21.15.5:0/2292324328 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd20803bd40 0x7fd20803e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd215ffb640 1 -- 172.21.15.5:0/2292324328 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (secure 0 0 0) 0x7fd2200ed920 con 0x7fd2100040b0 2024-09-13T11:40:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22896b640 1 --2- 172.21.15.5:0/2292324328 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd20803bd40 0x7fd20803e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.235+0000 7fd22896b640 10 monclient: get_auth_request con 0x7fd20803bd40 auth_method 0 2024-09-13T11:40:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.239+0000 7fd22896b640 1 --2- 172.21.15.5:0/2292324328 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd20803bd40 0x7fd20803e200 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd224074ad0 tx=0x7fd218011390 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.239+0000 7fd215ffb640 1 -- 172.21.15.5:0/2292324328 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fd220019c90 con 0x7fd2100040b0 2024-09-13T11:40:36.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.239+0000 7fd215ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:36.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.239+0000 7fd215ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:36.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.475+0000 7fd22b3d0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:40:36.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.475+0000 7fd22b3d0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:40:36.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.475+0000 7fd22b3d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:36.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.475+0000 7fd22b3d0640 1 -- 172.21.15.5:0/2292324328 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd224069ff0 con 0x7fd2100040b0 2024-09-13T11:40:36.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.475+0000 7fd215ffb640 1 -- 172.21.15.5:0/2292324328 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1542 (secure 0 0 0) 0x7fd22000cab0 con 0x7fd2100040b0 2024-09-13T11:40:36.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.475+0000 7fd215ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:40:36.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.475+0000 7fd215ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:36.478 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","modified":"2024-09-13T11:39:20.750174Z","created":"2024-09-13T11:39:20.750174Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:40:36.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 1 -- 172.21.15.5:0/2292324328 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd20803bd40 msgr2=0x7fd20803e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 1 --2- 172.21.15.5:0/2292324328 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd20803bd40 0x7fd20803e200 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd224074ad0 tx=0x7fd218011390 comp rx=0 tx=0).stop 2024-09-13T11:40:36.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 10 monclient: shutdown 2024-09-13T11:40:36.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:36.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 1 -- 172.21.15.5:0/2292324328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2100040b0 msgr2=0x7fd2241cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 1 --2- 172.21.15.5:0/2292324328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2100040b0 0x7fd2241cc620 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd220009100 tx=0x7fd210005840 comp rx=0 tx=0).stop 2024-09-13T11:40:36.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 1 -- 172.21.15.5:0/2292324328 shutdown_connections 2024-09-13T11:40:36.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 1 --2- 172.21.15.5:0/2292324328 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd20803bd40 0x7fd20803e200 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 1 --2- 172.21.15.5:0/2292324328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2240a1ae0 0x7fd224123250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 1 --2- 172.21.15.5:0/2292324328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd210004e20 0x7fd2241ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 1 --2- 172.21.15.5:0/2292324328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2100040b0 0x7fd2241cc620 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 1 -- 172.21.15.5:0/2292324328 >> 172.21.15.5:0/2292324328 conn(0x7fd2240a2930 msgr2=0x7fd22409e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:36.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 1 -- 172.21.15.5:0/2292324328 shutdown_connections 2024-09-13T11:40:36.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.479+0000 7fd22b3d0640 1 -- 172.21.15.5:0/2292324328 wait complete. 2024-09-13T11:40:36.495 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-13T11:40:36.495 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-13T11:40:36.495 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T11:40:36.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.611+0000 7efd1e60c640 10 monclient: get_monmap_and_config 2024-09-13T11:40:36.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.611+0000 7efd1e60c640 10 monclient: build_initial_monmap 2024-09-13T11:40:36.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.611+0000 7efd1e60c640 10 monclient: monmap: 2024-09-13T11:40:36.614 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:36.614 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.614 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:36.616351+0000 2024-09-13T11:40:36.614 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:36.616351+0000 2024-09-13T11:40:36.614 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:36.614 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.615 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:36.615 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:36.615 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:36.615 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:36.615 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:36.615 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 1 Processor -- start 2024-09-13T11:40:36.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 1 -- start start 2024-09-13T11:40:36.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 10 monclient: init 2024-09-13T11:40:36.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:36.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-13T11:40:36.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efd18134810 0x7efd18134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 10 monclient(hunting): picked mon.noname-a con 0x7efd18134810 addr v2:172.21.15.5:3300/0 2024-09-13T11:40:36.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd1809d1d0 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:36.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd1809dcf0 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:36.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd1809ce60 con 0x7efd1809d1d0 2024-09-13T11:40:36.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd1809eeb0 con 0x7efd1809dcf0 2024-09-13T11:40:36.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7efd180a0950 con 0x7efd18134810 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 10 monclient(hunting): authenticate will time out at 1039.965122s 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efd18134810 0x7efd18134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1cba9640 1 -- >> v1:172.21.15.39:6789/0 conn(0x7efd1809dcf0 legacy=0x7efd1813c690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.39:6789/0 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd0ffff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7efd1809d1d0 0x7efd1809d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:44692/0 (socket says 172.21.15.5:44692) 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efd18134810 0x7efd18134bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49720/0 (socket says 172.21.15.5:49720) 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd0ffff640 1 -- 172.21.15.5:0/1353414853 learned_addr learned my addr 172.21.15.5:0/1353414853 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 10 monclient(hunting): get_auth_request con 0x7efd18134810 auth_method 0 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:36.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 10 monclient(hunting): handle_auth_done global_id 4512 payload 274 2024-09-13T11:40:36.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 1 -- 172.21.15.5:0/1353414853 >> v1:172.21.15.39:6789/0 conn(0x7efd1809dcf0 legacy=0x7efd1813c690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:36.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 1 -- 172.21.15.5:0/1353414853 >> v1:172.21.15.5:6789/0 conn(0x7efd1809d1d0 legacy=0x7efd1809d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 10 monclient: _finish_hunting 0 2024-09-13T11:40:36.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 1 monclient: found mon.noname-a 2024-09-13T11:40:36.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:36.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd177fe640 1 -- 172.21.15.5:0/1353414853 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1052571680 0 0) 0x7efd1809ce60 con 0x7efd1809d1d0 2024-09-13T11:40:36.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:36.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 1 -- 172.21.15.5:0/1353414853 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd18135130 con 0x7efd18134810 2024-09-13T11:40:36.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd177fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:40:36.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd17fff640 1 --2- 172.21.15.5:0/1353414853 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd18134810 0x7efd18134bf0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efd080097c0 tx=0x7efd0801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4dc1f906bcbde92 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd177fe640 1 -- 172.21.15.5:0/1353414853 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd080b0070 con 0x7efd18134810 2024-09-13T11:40:36.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:36.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd177fe640 10 monclient: dump: 2024-09-13T11:40:36.625 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.625 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd177fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd177fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd177fe640 1 -- 172.21.15.5:0/1353414853 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd080ab600 con 0x7efd18134810 2024-09-13T11:40:36.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.615+0000 7efd1e60c640 5 monclient: authenticate success, global_id 4512 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd177fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 4 set_mon_vals no callback set 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd177fe640 1 -- 172.21.15.5:0/1353414853 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd080ab9d0 con 0x7efd18134810 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd177fe640 10 monclient: dump: 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.627 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient: shutdown 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 -- 172.21.15.5:0/1353414853 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd18134810 msgr2=0x7efd18134bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 --2- 172.21.15.5:0/1353414853 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd18134810 0x7efd18134bf0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efd080097c0 tx=0x7efd0801c360 comp rx=0 tx=0).stop 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 -- 172.21.15.5:0/1353414853 shutdown_connections 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 --2- 172.21.15.5:0/1353414853 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd18134810 0x7efd18134bf0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 -- 172.21.15.5:0/1353414853 >> 172.21.15.5:0/1353414853 conn(0x7efd180a2930 msgr2=0x7efd180a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 -- 172.21.15.5:0/1353414853 shutdown_connections 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 -- 172.21.15.5:0/1353414853 wait complete. 2024-09-13T11:40:36.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient: build_initial_monmap 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient: monmap: 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 Processor -- start 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 -- start start 2024-09-13T11:40:36.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient: init 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efd1809d1d0 0x7efd18132cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient(hunting): picked mon.noname-c con 0x7efd1809d1d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd1809dcf0 0x7efd18133200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient(hunting): picked mon.noname-b con 0x7efd1809dcf0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd18134810 0x7efd1813b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient(hunting): picked mon.noname-a con 0x7efd18134810 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7efd1809e370 con 0x7efd18134810 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7efd180a0d70 con 0x7efd1809dcf0 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7efd181405d0 con 0x7efd1809d1d0 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient(hunting): authenticate will time out at 1039.970305s 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd18134810 0x7efd1813b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efd1809d1d0 0x7efd18132cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd0ffff640 1 -- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd1809dcf0 msgr2=0x7efd18133200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efd1809d1d0 0x7efd18132cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47106/0 (socket says 172.21.15.5:47106) 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 1 -- 172.21.15.5:0/955560431 learned_addr learned my addr 172.21.15.5:0/955560431 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd18134810 0x7efd1813b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49726/0 (socket says 172.21.15.5:49726) 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 10 monclient(hunting): get_auth_request con 0x7efd1809d1d0 auth_method 0 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd0ffff640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd1809dcf0 0x7efd18133200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 10 monclient(hunting): get_auth_request con 0x7efd18134810 auth_method 0 2024-09-13T11:40:36.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:36.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:36.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 10 monclient(hunting): handle_auth_done global_id 4518 payload 824 2024-09-13T11:40:36.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 1 -- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efd1809d1d0 msgr2=0x7efd18132cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efd1809d1d0 0x7efd18132cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 1 -- 172.21.15.5:0/955560431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd1809dcf0 msgr2=0x7efd18133200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd1809dcf0 0x7efd18133200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 10 monclient: _finish_hunting 0 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 1 monclient: found mon.noname-a 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 1 -- 172.21.15.5:0/955560431 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd1811d6f0 con 0x7efd18134810 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1cba9640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd18134810 0x7efd1813b1d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efd1000ba60 tx=0x7efd1000bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1000cbe0 con 0x7efd18134810 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 10 monclient: dump: 2024-09-13T11:40:36.633 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 10 monclient: _finish_auth 0 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 10 monclient: _check_auth_tickets 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 5 monclient: authenticate success, global_id 4518 2024-09-13T11:40:36.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd10010070 con 0x7efd18134810 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient: _renew_subs 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 -- 172.21.15.5:0/955560431 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd1811d9d0 con 0x7efd18134810 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd10015470 con 0x7efd18134810 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd157fa640 10 monclient: dump: 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.635 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd16ffd640 4 set_mon_vals no callback set 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient: _renew_subs 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 -- 172.21.15.5:0/955560431 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd18073d80 con 0x7efd18134810 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813b710 0x7efd1813baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd1e60c640 1 -- 172.21.15.5:0/955560431 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7efd18134bf0 con 0x7efd1813b710 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813b710 0x7efd1813baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 10 monclient: get_auth_request con 0x7efd1813b710 auth_method 0 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.619+0000 7efd17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd17fff640 10 monclient(hunting): handle_auth_done global_id 4521 payload 824 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd17fff640 1 -- 172.21.15.5:0/955560431 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd180008e0 con 0x7efd1813b710 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7efd10015610 con 0x7efd18134810 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd17fff640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813b710 0x7efd1813baf0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efd080ae040 tx=0x7efd0801aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 1 --2- 172.21.15.5:0/955560431 >> v2:172.21.15.39:6812/1340838781 conn(0x7efcf803b280 0x7efcf803d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (secure 0 0 0) 0x7efd1004ee10 con 0x7efd18134810 2024-09-13T11:40:36.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd080b0040 con 0x7efd1813b710 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 10 monclient: dump: 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efd080b6640 con 0x7efd1813b710 2024-09-13T11:40:36.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:36.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:36.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813b710 msgr2=0x7efd1813baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813b710 0x7efd1813baf0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efd080ae040 tx=0x7efd0801aba0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813b710 msgr2=0x7efd1813baf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:40:36.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd157fa640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813b710 0x7efd1813baf0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd0ffff640 1 --2- 172.21.15.5:0/955560431 >> v2:172.21.15.39:6812/1340838781 conn(0x7efcf803b280 0x7efcf803d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd0ffff640 10 monclient: get_auth_request con 0x7efcf803b280 auth_method 0 2024-09-13T11:40:36.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.623+0000 7efd0ffff640 1 --2- 172.21.15.5:0/955560431 >> v2:172.21.15.39:6812/1340838781 conn(0x7efcf803b280 0x7efcf803d740 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efd000010f0 tx=0x7efd00007a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd1e60c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:40:36.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd1e60c640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813c030 0x7efd18137450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd1e60c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd1e60c640 1 -- 172.21.15.5:0/955560431 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7efd1811d410 con 0x7efd1813c030 2024-09-13T11:40:36.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd17fff640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813c030 0x7efd18137450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd17fff640 10 monclient: get_auth_request con 0x7efd1813c030 auth_method 0 2024-09-13T11:40:36.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd17fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd17fff640 10 monclient(hunting): handle_auth_done global_id 4524 payload 824 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd17fff640 1 -- 172.21.15.5:0/955560431 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efd18133740 con 0x7efd1813c030 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd17fff640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813c030 0x7efd18137450 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efd080ac000 tx=0x7efd08019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd080b0050 con 0x7efd1813c030 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd157fa640 10 monclient: dump: 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.642 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4857 (secure 0 0 0) 0x7efd08003830 con 0x7efd1813c030 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd157fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd157fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813c030 msgr2=0x7efd18137450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd157fa640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813c030 0x7efd18137450 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efd080ac000 tx=0x7efd08019100 comp rx=0 tx=0).stop 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd157fa640 1 -- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813c030 msgr2=0x7efd18137450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:40:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.627+0000 7efd157fa640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd1813c030 0x7efd18137450 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efd080ac000 tx=0x7efd08019100 comp rx=0 tx=0).stop 2024-09-13T11:40:36.644 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:40:36.644 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-13T11:40:36.644 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-13T11:40:36.644 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-13T11:40:36.644 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 12, 2024-09-13T11:40:36.644 INFO:teuthology.orchestra.run.smithi005.stdout: "uptime": 48599, 2024-09-13T11:40:36.644 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-13T11:40:36.644 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-13T11:40:36.644 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-13T11:40:36.644 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:36.644 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2450521708007866372", 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:36.645 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:36.646 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "59046471-79a9-4d90-9616-da0b446053a7", 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 19, 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "squid", 2024-09-13T11:40:36.647 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 1, 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:36.648 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.649 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:36.650 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:36.651 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:36.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.651 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-13T11:40:36.651 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-13T11:40:36.651 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:36.651 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:36.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.651 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:36.651 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:3300", 2024-09-13T11:40:36.651 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:36.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789", 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789/0", 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.39:6789/0", 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.653 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:36.654 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:36.655 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-13T11:40:36.656 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:40:36.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.631+0000 7efd1e60c640 1 -- 172.21.15.5:0/955560431 >> v2:172.21.15.39:6812/1340838781 conn(0x7efcf803b280 msgr2=0x7efcf803d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.631+0000 7efd1e60c640 1 --2- 172.21.15.5:0/955560431 >> v2:172.21.15.39:6812/1340838781 conn(0x7efcf803b280 0x7efcf803d740 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efd000010f0 tx=0x7efd00007a90 comp rx=0 tx=0).stop 2024-09-13T11:40:36.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.631+0000 7efd1e60c640 10 monclient: shutdown 2024-09-13T11:40:36.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.631+0000 7efd1e60c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:36.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.631+0000 7efd1e60c640 1 -- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd18134810 msgr2=0x7efd1813b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.631+0000 7efd1cba9640 1 -- 172.21.15.5:0/955560431 reap_dead start 2024-09-13T11:40:36.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.631+0000 7efd1e60c640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd18134810 0x7efd1813b1d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efd1000ba60 tx=0x7efd1000bf20 comp rx=0 tx=0).stop 2024-09-13T11:40:36.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.631+0000 7efd1e60c640 1 -- 172.21.15.5:0/955560431 shutdown_connections 2024-09-13T11:40:36.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.631+0000 7efd1e60c640 1 --2- 172.21.15.5:0/955560431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efd18134810 0x7efd1813b1d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.631+0000 7efd1e60c640 1 -- 172.21.15.5:0/955560431 >> 172.21.15.5:0/955560431 conn(0x7efd180a2930 msgr2=0x7efd1813d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:36.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.631+0000 7efd1e60c640 1 -- 172.21.15.5:0/955560431 shutdown_connections 2024-09-13T11:40:36.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.631+0000 7efd1e60c640 1 -- 172.21.15.5:0/955560431 wait complete. 2024-09-13T11:40:36.659 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T11:40:36.659 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-13T11:40:36.659 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T11:40:36.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 10 monclient: get_monmap_and_config 2024-09-13T11:40:36.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 10 monclient: build_initial_monmap 2024-09-13T11:40:36.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 10 monclient: monmap: 2024-09-13T11:40:36.779 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:36.779 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.779 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:40:36.781255+0000 2024-09-13T11:40:36.779 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:40:36.781255+0000 2024-09-13T11:40:36.779 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:36.780 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.780 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:40:36.780 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:40:36.780 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:40:36.780 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:40:36.780 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:40:36.780 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 1 Processor -- start 2024-09-13T11:40:36.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 1 -- start start 2024-09-13T11:40:36.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 10 monclient: init 2024-09-13T11:40:36.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:36.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-13T11:40:36.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f40137930 addr v1:172.21.15.5:6789/0 2024-09-13T11:40:36.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f40138420 addr v1:172.21.15.39:6789/0 2024-09-13T11:40:36.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2f4009cc90 0x7f2f401404b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 10 monclient(hunting): picked mon.noname-c con 0x7f2f4009cc90 addr v2:172.21.15.39:3300/0 2024-09-13T11:40:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f400a5a80 con 0x7f2f40137930 2024-09-13T11:40:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f4009bef0 con 0x7f2f40138420 2024-09-13T11:40:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f2f400a5fe0 con 0x7f2f4009cc90 2024-09-13T11:40:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 10 monclient(hunting): authenticate will time out at 1040.130185s 2024-09-13T11:40:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 1 -- >> v2:172.21.15.39:3300/0 conn(0x7f2f4009cc90 msgr2=0x7f2f401404b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3e7fc640 1 -- >> v1:172.21.15.39:6789/0 conn(0x7f2f40138420 legacy=0x7f2f4013cd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.39:6789/0 2024-09-13T11:40:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3effd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f2f40137930 0x7f2f40137d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:44698/0 (socket says 172.21.15.5:44698) 2024-09-13T11:40:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3effd640 1 -- 172.21.15.5:0/716928308 learned_addr learned my addr 172.21.15.5:0/716928308 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:40:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 1 --2- 172.21.15.5:0/716928308 >> v2:172.21.15.39:3300/0 conn(0x7f2f4009cc90 0x7f2f401404b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2795412799 0 0) 0x7f2f400a5a80 con 0x7f2f40137930 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient(hunting): _negotiate 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient(hunting): my global_id is 4527 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f28003530 con 0x7f2f40137930 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 510478475 0 0) 0x7f2f28003530 con 0x7f2f40137930 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 >> v2:172.21.15.39:3300/0 conn(0x7f2f4009cc90 msgr2=0x7f2f401404b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 --2- 172.21.15.5:0/716928308 >> v2:172.21.15.39:3300/0 conn(0x7f2f4009cc90 0x7f2f401404b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 >> v1:172.21.15.39:6789/0 conn(0x7f2f40138420 legacy=0x7f2f4013cd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient: _finish_hunting 0 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:40:36.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4009c1d0 con 0x7f2f40137930 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient: _finish_auth 0 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 5 monclient: authenticate success, global_id 4527 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3199362888 0 0) 0x7f2f34001700 con 0x7f2f40137930 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient: dump: 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 >> v1:172.21.15.5:6789/0 conn(0x7f2f40137930 legacy=0x7f2f40137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 --2- 172.21.15.5:0/716928308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f280039c0 0x7f2f28003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient(hunting): picked mon.b con 0x7f2f280039c0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 --2- 172.21.15.5:0/716928308 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f28004810 0x7f2f28004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient(hunting): picked mon.a con 0x7f2f28004810 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 --2- 172.21.15.5:0/716928308 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f28005380 0x7f2f2800b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient(hunting): picked mon.c con 0x7f2f28005380 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2800c2a0 con 0x7f2f28004810 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2800c450 con 0x7f2f280039c0 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f2f2800c650 con 0x7f2f28005380 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3dffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f4570a640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3effd640 1 -- 172.21.15.5:0/716928308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f280039c0 msgr2=0x7f2f28003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:36.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3e7fc640 1 --2- 172.21.15.5:0/716928308 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f28004810 0x7f2f28004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3effd640 1 --2- 172.21.15.5:0/716928308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f280039c0 0x7f2f28003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3e7fc640 10 monclient(hunting): get_auth_request con 0x7f2f28004810 auth_method 0 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 1 --2- 172.21.15.5:0/716928308 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f28005380 0x7f2f2800b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 10 monclient(hunting): get_auth_request con 0x7f2f28005380 auth_method 0 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 10 monclient(hunting): handle_auth_done global_id 4527 payload 293 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 1 -- 172.21.15.5:0/716928308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f280039c0 msgr2=0x7f2f28003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 1 --2- 172.21.15.5:0/716928308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f280039c0 0x7f2f28003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 1 -- 172.21.15.5:0/716928308 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f28004810 msgr2=0x7f2f28004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 1 --2- 172.21.15.5:0/716928308 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f28004810 0x7f2f28004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 1 monclient: found mon.c 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 1 -- 172.21.15.5:0/716928308 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f28003530 con 0x7f2f28005380 2024-09-13T11:40:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.779+0000 7f2f3e7fc640 1 --2- 172.21.15.5:0/716928308 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f28004810 0x7f2f28004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f3f7fe640 1 -- 172.21.15.5:0/716928308 reap_dead start 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f3f7fe640 1 --2- 172.21.15.5:0/716928308 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f28005380 0x7f2f2800b9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009e30 tx=0x7f2f3801caf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=937f9d94f088ed5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30009520 con 0x7f2f28005380 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f3dffb640 10 monclient: dump: 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f380a9070 con 0x7f2f28005380 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f3dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 10 monclient: get_monmap_and_config success 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f3dffb640 1 -- 172.21.15.5:0/716928308 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38009a30 con 0x7f2f28005380 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 4 set_mon_vals no callback set 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f3dffb640 10 monclient: dump: 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 10 monclient: shutdown 2024-09-13T11:40:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 1 -- 172.21.15.5:0/716928308 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f28005380 msgr2=0x7f2f2800b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 1 --2- 172.21.15.5:0/716928308 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f28005380 0x7f2f2800b9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009e30 tx=0x7f2f3801caf0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 1 -- 172.21.15.5:0/716928308 shutdown_connections 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 1 --2- 172.21.15.5:0/716928308 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f28005380 0x7f2f2800b9f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 1 -- 172.21.15.5:0/716928308 >> 172.21.15.5:0/716928308 conn(0x7f2f400a6c70 msgr2=0x7f2f400a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 1 -- 172.21.15.5:0/716928308 shutdown_connections 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 1 -- 172.21.15.5:0/716928308 wait complete. 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 10 monclient: build_initial_monmap 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 10 monclient: monmap: 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:40:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 1 Processor -- start 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 1 -- start start 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.783+0000 7f2f4570a640 10 monclient: init 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient: _reopen_session rank -1 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f28005380 0x7f2f4013ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient(hunting): picked mon.noname-a con 0x7f2f28005380 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:40:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f4013cce0 0x7f2f40136a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient(hunting): picked mon.noname-c con 0x7f2f4013cce0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f401cf510 0x7f2f401cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient(hunting): picked mon.noname-b con 0x7f2f401cf510 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2f401380a0 con 0x7f2f28005380 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2f40137f20 con 0x7f2f401cf510 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f2f40138220 con 0x7f2f4013cce0 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient(hunting): _renew_subs 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient(hunting): authenticate will time out at 1040.136291s 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f4013cce0 0x7f2f40136a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3f7fe640 1 -- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f401cf510 msgr2=0x7f2f401cf970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:3300/0 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3f7fe640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f401cf510 0x7f2f401cf970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f4013cce0 0x7f2f40136a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47126/0 (socket says 172.21.15.5:47126) 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f28005380 0x7f2f4013ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 1 -- 172.21.15.5:0/2865195947 learned_addr learned my addr 172.21.15.5:0/2865195947 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 10 monclient(hunting): get_auth_request con 0x7f2f4013cce0 auth_method 0 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): get_auth_request con 0x7f2f28005380 auth_method 0 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 10 monclient(hunting): handle_auth_done global_id 4379 payload 824 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 1 -- 172.21.15.5:0/2865195947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f401cf510 msgr2=0x7f2f401cf970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f401cf510 0x7f2f401cf970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 1 -- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f28005380 msgr2=0x7f2f4013ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2f28005380 0x7f2f4013ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 10 monclient: _finish_hunting 0 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 1 monclient: found mon.noname-c 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:40:36.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 1 -- 172.21.15.5:0/2865195947 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4013f790 con 0x7f2f4013cce0 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3e7fc640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f4013cce0 0x7f2f40136a60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2f3000e630 tx=0x7f2f3000eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30016020 con 0x7f2f4013cce0 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: dump: 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.808 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.808 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: _finish_auth 0 2024-09-13T11:40:36.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: _check_auth_tickets 2024-09-13T11:40:36.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:40:36.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 5 monclient: authenticate success, global_id 4379 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f30003bd0 con 0x7f2f4013cce0 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f300050d0 con 0x7f2f4013cce0 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient: _renew_subs 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 1 -- 172.21.15.5:0/2865195947 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f4013d100 con 0x7f2f4013cce0 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3d7fa640 4 set_mon_vals no callback set 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: dump: 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient: _renew_subs 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 1 -- 172.21.15.5:0/2865195947 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f4013e2a0 con 0x7f2f4013cce0 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f2f30019020 con 0x7f2f4013cce0 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401cfeb0 0x7f2f40138d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f4570a640 1 -- 172.21.15.5:0/2865195947 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f2f400a1360 con 0x7f2f401cfeb0 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401cfeb0 0x7f2f40138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient: get_auth_request con 0x7f2f401cfeb0 auth_method 0 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 1 --2- 172.21.15.5:0/2865195947 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2f2c03b250 0x7f2f2c03d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (secure 0 0 0) 0x7f2f3004eb50 con 0x7f2f4013cce0 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 10 monclient(hunting): handle_auth_done global_id 4382 payload 824 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 1 -- 172.21.15.5:0/2865195947 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f400a4c20 con 0x7f2f401cfeb0 2024-09-13T11:40:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3effd640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401cfeb0 0x7f2f40138d60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400f8b0 tx=0x7f2f3400fd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3f7fe640 1 --2- 172.21.15.5:0/2865195947 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2f2c03b250 0x7f2f2c03d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3f7fe640 10 monclient: get_auth_request con 0x7f2f2c03b250 auth_method 0 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34019070 con 0x7f2f401cfeb0 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: dump: 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.813 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.813 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2f3400d910 con 0x7f2f401cfeb0 2024-09-13T11:40:36.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:40:36.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:40:36.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401cfeb0 msgr2=0x7f2f40138d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401cfeb0 0x7f2f40138d60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400f8b0 tx=0x7f2f3400fd70 comp rx=0 tx=0).stop 2024-09-13T11:40:36.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401cfeb0 msgr2=0x7f2f40138d60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:40:36.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f277fe640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401cfeb0 0x7f2f40138d60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.787+0000 7f2f3f7fe640 1 --2- 172.21.15.5:0/2865195947 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2f2c03b250 0x7f2f2c03d710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800c820 tx=0x7f2f38005e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:40:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401392a0 0x7f2f401396b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:40:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:40:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 1 -- 172.21.15.5:0/2865195947 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f2f401d0290 con 0x7f2f401392a0 2024-09-13T11:40:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3effd640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401392a0 0x7f2f401396b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:40:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3effd640 10 monclient: get_auth_request con 0x7f2f401392a0 auth_method 0 2024-09-13T11:40:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:40:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3effd640 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3effd640 1 -- 172.21.15.5:0/2865195947 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2f4006cbe0 con 0x7f2f401392a0 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3effd640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401392a0 0x7f2f401396b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2f34016000 tx=0x7f2f34001070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34019040 con 0x7f2f401392a0 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f277fe640 10 monclient: dump: 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:40:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4688 (secure 0 0 0) 0x7f2f34008070 con 0x7f2f401392a0 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f277fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f277fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401392a0 msgr2=0x7f2f401396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f277fe640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401392a0 0x7f2f401396b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2f34016000 tx=0x7f2f34001070 comp rx=0 tx=0).stop 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f277fe640 1 -- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401392a0 msgr2=0x7f2f401396b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:40:36.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f277fe640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f401392a0 0x7f2f401396b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.818 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:40:36.818 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-13T11:40:36.818 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-13T11:40:36.818 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-13T11:40:36.818 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 12, 2024-09-13T11:40:36.818 INFO:teuthology.orchestra.run.smithi005.stdout: "uptime": 48767, 2024-09-13T11:40:36.818 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-13T11:40:36.818 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-13T11:40:36.818 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-13T11:40:36.818 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2450521708007866372", 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:36.819 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:36.820 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "59046471-79a9-4d90-9616-da0b446053a7", 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:40:36.821 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 19, 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "squid", 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 1, 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:40:36.822 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:40:36.823 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:40:36.823 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:40:36.823 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:40:36.823 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:40:36.823 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:40:36.823 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:36.823 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-13T11:40:36.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.823 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-13T11:40:36.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.823 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-13T11:40:36.823 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.824 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:36.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:3300", 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789", 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789/0", 2024-09-13T11:40:36.826 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.39:6789/0", 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-13T11:40:36.827 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:40:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-13T11:40:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 6 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:40:36.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:40:36.831 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-13T11:40:36.831 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:40:36.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 1 -- 172.21.15.5:0/2865195947 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2f2c03b250 msgr2=0x7f2f2c03d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 1 --2- 172.21.15.5:0/2865195947 >> v2:172.21.15.39:6812/1340838781 conn(0x7f2f2c03b250 0x7f2f2c03d710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800c820 tx=0x7f2f38005e10 comp rx=0 tx=0).stop 2024-09-13T11:40:36.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 10 monclient: shutdown 2024-09-13T11:40:36.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:40:36.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 1 -- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f4013cce0 msgr2=0x7f2f40136a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:40:36.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 1 --2- 172.21.15.5:0/2865195947 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2f4013cce0 0x7f2f40136a60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2f3000e630 tx=0x7f2f3000eaf0 comp rx=0 tx=0).stop 2024-09-13T11:40:36.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3f7fe640 1 -- 172.21.15.5:0/2865195947 reap_dead start 2024-09-13T11:40:36.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f3f7fe640 1 -- 172.21.15.5:0/2865195947 reap_dead start 2024-09-13T11:40:36.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 1 -- 172.21.15.5:0/2865195947 shutdown_connections 2024-09-13T11:40:36.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 1 -- 172.21.15.5:0/2865195947 >> 172.21.15.5:0/2865195947 conn(0x7f2f400a6c70 msgr2=0x7f2f4009fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:40:36.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 1 -- 172.21.15.5:0/2865195947 shutdown_connections 2024-09-13T11:40:36.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:40:36.795+0000 7f2f4570a640 1 -- 172.21.15.5:0/2865195947 wait complete. 2024-09-13T11:40:36.833 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T11:40:36.836 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-13T11:40:37.410 INFO:teuthology.orchestra.run.smithi039.stdout:process_1_[27760]: starting. 2024-09-13T11:40:37.411 INFO:teuthology.orchestra.run.smithi039.stdout:process_1_[27760]: creating pool ceph_test_rados_delete_pools_parallel.smithi039-27759 2024-09-13T11:40:37.411 INFO:teuthology.orchestra.run.smithi039.stdout:process_1_[27760]: created object 0... 2024-09-13T11:40:37.411 INFO:teuthology.orchestra.run.smithi039.stdout:process_1_[27760]: created object 25... 2024-09-13T11:40:37.411 INFO:teuthology.orchestra.run.smithi039.stdout:process_1_[27760]: created object 49... 2024-09-13T11:40:37.411 INFO:teuthology.orchestra.run.smithi039.stdout:process_1_[27760]: finishing. 2024-09-13T11:40:37.411 INFO:teuthology.orchestra.run.smithi039.stdout:process_1_[27760]: shutting down. 2024-09-13T11:40:38.221 INFO:teuthology.orchestra.run.smithi039.stdout:process_2_[27761]: starting. 2024-09-13T11:40:38.221 INFO:teuthology.orchestra.run.smithi039.stdout:process_2_[27761]: deleting pool ceph_test_rados_delete_pools_parallel.smithi039-27759 2024-09-13T11:40:38.221 INFO:teuthology.orchestra.run.smithi039.stdout:process_2_[27761]: shutting down. 2024-09-13T11:40:40.435 INFO:teuthology.orchestra.run.smithi039.stdout:******************************* 2024-09-13T11:40:40.436 INFO:teuthology.orchestra.run.smithi039.stdout:process_3_[27794]: starting. 2024-09-13T11:40:40.436 INFO:teuthology.orchestra.run.smithi039.stdout:process_3_[27794]: creating pool ceph_test_rados_delete_pools_parallel.smithi039-27759 2024-09-13T11:40:40.436 INFO:teuthology.orchestra.run.smithi039.stdout:process_3_[27794]: created object 0... 2024-09-13T11:40:40.436 INFO:teuthology.orchestra.run.smithi039.stdout:process_3_[27794]: created object 25... 2024-09-13T11:40:40.436 INFO:teuthology.orchestra.run.smithi039.stdout:process_3_[27794]: created object 49... 2024-09-13T11:40:40.436 INFO:teuthology.orchestra.run.smithi039.stdout:process_3_[27794]: finishing. 2024-09-13T11:40:40.436 INFO:teuthology.orchestra.run.smithi039.stdout:process_3_[27794]: shutting down. 2024-09-13T11:40:40.471 INFO:teuthology.orchestra.run.smithi039.stdout:******************************* 2024-09-13T11:40:40.471 INFO:teuthology.orchestra.run.smithi039.stdout:process_5_[27796]: starting. 2024-09-13T11:40:40.471 INFO:teuthology.orchestra.run.smithi039.stdout:process_5_[27796]: listing objects. 2024-09-13T11:40:40.471 INFO:teuthology.orchestra.run.smithi039.stdout:process_5_[27796]: listed object 0... 2024-09-13T11:40:40.471 INFO:teuthology.orchestra.run.smithi039.stdout:process_5_[27796]: listed object 25... 2024-09-13T11:40:40.471 INFO:teuthology.orchestra.run.smithi039.stdout:process_5_[27796]: saw 50 objects 2024-09-13T11:40:40.472 INFO:teuthology.orchestra.run.smithi039.stdout:process_5_[27796]: shutting down. 2024-09-13T11:40:41.257 INFO:teuthology.orchestra.run.smithi039.stdout:******************************* 2024-09-13T11:40:41.257 INFO:teuthology.orchestra.run.smithi039.stdout:process_4_[27795]: starting. 2024-09-13T11:40:41.257 INFO:teuthology.orchestra.run.smithi039.stdout:process_4_[27795]: deleting pool ceph_test_rados_delete_pools_parallel.smithi039-27759 2024-09-13T11:40:41.257 INFO:teuthology.orchestra.run.smithi039.stdout:process_4_[27795]: shutting down. 2024-09-13T11:40:41.260 INFO:teuthology.orchestra.run.smithi039.stdout:******************************* 2024-09-13T11:40:41.261 INFO:teuthology.orchestra.run.smithi039.stdout:******************************* 2024-09-13T11:40:41.261 INFO:teuthology.orchestra.run.smithi039.stdout:******* SUCCESS ********** 2024-09-13T11:40:41.264 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-13T11:40:41.276 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-13T11:42:06.837 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-13T11:42:06.838 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T11:42:06.839 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T11:42:06.842 INFO:tasks.ceph.mon.b:Started 2024-09-13T11:42:06.843 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T11:42:06.843 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:42:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient: get_monmap_and_config 2024-09-13T11:42:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient: build_initial_monmap 2024-09-13T11:42:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient: monmap: 2024-09-13T11:42:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:06.978105+0000 2024-09-13T11:42:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:06.978105+0000 2024-09-13T11:42:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:06.982 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:06.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 1 Processor -- start 2024-09-13T11:42:06.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 1 -- start start 2024-09-13T11:42:06.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient: init 2024-09-13T11:42:06.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:06.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-13T11:42:06.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1f28137ac0 0x7f1f2809cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:06.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient(hunting): picked mon.noname-c con 0x7f1f28137ac0 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:06.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1f28137ea0 0x7f1f280a2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:06.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient(hunting): picked mon.noname-a con 0x7f1f28137ea0 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f281382f0 0x7f1f280a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient(hunting): picked mon.noname-b con 0x7f1f281382f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f1f2809c4d0 con 0x7f1f28137ea0 2024-09-13T11:42:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f1f2806d790 con 0x7f1f28137ac0 2024-09-13T11:42:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f1f28074f80 con 0x7f1f281382f0 2024-09-13T11:42:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2e4c2640 10 monclient(hunting): authenticate will time out at 1130.327610s 2024-09-13T11:42:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f1ffff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1f28137ea0 0x7f1f280a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f1ffff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1f28137ea0 0x7f1f280a2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52542/0 (socket says 172.21.15.5:52542) 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f1ffff640 1 -- 172.21.15.5:0/2819413719 learned_addr learned my addr 172.21.15.5:0/2819413719 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f1ffff640 10 monclient(hunting): get_auth_request con 0x7f1f28137ea0 auth_method 0 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2ca5f640 1 --2- 172.21.15.5:0/2819413719 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f281382f0 0x7f1f280a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2ca5f640 10 monclient(hunting): get_auth_request con 0x7f1f281382f0 auth_method 0 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2ca5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2ca5f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.975+0000 7f1f2ca5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f27fff640 1 --2- 172.21.15.5:0/2819413719 >> v2:172.21.15.39:3300/0 conn(0x7f1f28137ac0 0x7f1f2809cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f27fff640 10 monclient(hunting): get_auth_request con 0x7f1f28137ac0 auth_method 0 2024-09-13T11:42:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f27fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 10 monclient(hunting): handle_auth_done global_id 4403 payload 274 2024-09-13T11:42:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 1 -- 172.21.15.5:0/2819413719 >> v2:172.21.15.39:3300/0 conn(0x7f1f28137ac0 msgr2=0x7f1f2809cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 1 --2- 172.21.15.5:0/2819413719 >> v2:172.21.15.39:3300/0 conn(0x7f1f28137ac0 0x7f1f2809cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 1 -- 172.21.15.5:0/2819413719 >> v2:172.21.15.5:3300/0 conn(0x7f1f28137ea0 msgr2=0x7f1f280a2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 1 --2- 172.21.15.5:0/2819413719 >> v2:172.21.15.5:3300/0 conn(0x7f1f28137ea0 0x7f1f280a2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 10 monclient: _finish_hunting 0 2024-09-13T11:42:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 1 monclient: found mon.noname-b 2024-09-13T11:42:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 1 -- 172.21.15.5:0/2819413719 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f2806d310 con 0x7f1f281382f0 2024-09-13T11:42:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f1ffff640 1 --2- 172.21.15.5:0/2819413719 >> v2:172.21.15.5:3300/0 conn(0x7f1f28137ea0 0x7f1f280a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2ca5f640 1 --2- 172.21.15.5:0/2819413719 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f281382f0 0x7f1f280a2e40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1f280a4850 tx=0x7f1f2001c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20f6b34d4510e05e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 1 -- 172.21.15.5:0/2819413719 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f200b2070 con 0x7f1f281382f0 2024-09-13T11:42:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:42:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 10 monclient: dump: 2024-09-13T11:42:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 10 monclient: _finish_auth 0 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 10 monclient: _check_auth_tickets 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 5 monclient: authenticate success, global_id 4403 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 1 -- 172.21.15.5:0/2819413719 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f200a5810 con 0x7f1f281382f0 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 1 -- 172.21.15.5:0/2819413719 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f200a5be0 con 0x7f1f281382f0 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f277fe640 10 monclient: dump: 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.006 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.006 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.006 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.006 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:07.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 4 set_mon_vals no callback set 2024-09-13T11:42:07.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 10 monclient: shutdown 2024-09-13T11:42:07.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:07.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/2819413719 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f281382f0 msgr2=0x7f1f280a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 1 --2- 172.21.15.5:0/2819413719 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f281382f0 0x7f1f280a2e40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1f280a4850 tx=0x7f1f2001c880 comp rx=0 tx=0).stop 2024-09-13T11:42:07.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/2819413719 shutdown_connections 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 1 --2- 172.21.15.5:0/2819413719 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f281382f0 0x7f1f280a2e40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 1 --2- 172.21.15.5:0/2819413719 >> v2:172.21.15.5:3300/0 conn(0x7f1f28137ea0 0x7f1f280a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 1 --2- 172.21.15.5:0/2819413719 >> v2:172.21.15.39:3300/0 conn(0x7f1f28137ac0 0x7f1f2809cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/2819413719 >> 172.21.15.5:0/2819413719 conn(0x7f1f280a39f0 msgr2=0x7f1f280a3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/2819413719 shutdown_connections 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/2819413719 wait complete. 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 10 monclient: build_initial_monmap 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.979+0000 7f1f2e4c2640 10 monclient: monmap: 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:07.007 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 1 Processor -- start 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 1 -- start start 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient: init 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f28137ac0 0x7f1f281cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient(hunting): picked mon.noname-b con 0x7f1f28137ac0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f28137ea0 0x7f1f2811ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient(hunting): picked mon.noname-a con 0x7f1f28137ea0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f281382f0 0x7f1f28134870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient(hunting): picked mon.noname-c con 0x7f1f281382f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1f281413d0 con 0x7f1f28137ea0 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1f280a3790 con 0x7f1f28137ac0 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f1f28141550 con 0x7f1f281382f0 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient(hunting): authenticate will time out at 1130.332823s 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f28137ea0 0x7f1f2811ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f28137ea0 0x7f1f2811ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52546/0 (socket says 172.21.15.5:52546) 2024-09-13T11:42:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2ca5f640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f281382f0 0x7f1f28134870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 1 -- 172.21.15.5:0/455292394 learned_addr learned my addr 172.21.15.5:0/455292394 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 10 monclient(hunting): get_auth_request con 0x7f1f28137ea0 auth_method 0 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2ca5f640 10 monclient(hunting): get_auth_request con 0x7f1f281382f0 auth_method 0 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2ca5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2ca5f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2ca5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2ca5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2ca5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f27fff640 1 --2- 172.21.15.5:0/455292394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f28137ac0 0x7f1f281cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2ca5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f27fff640 10 monclient(hunting): get_auth_request con 0x7f1f28137ac0 auth_method 0 2024-09-13T11:42:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f27fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f27fff640 1 -- 172.21.15.5:0/455292394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f28137ac0 msgr2=0x7f1f281cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T11:42:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f27fff640 1 -- 172.21.15.5:0/455292394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f28137ac0 msgr2=0x7f1f281cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:42:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f27fff640 1 --2- 172.21.15.5:0/455292394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f28137ac0 0x7f1f281cf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:42:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 10 monclient(hunting): handle_auth_done global_id 4551 payload 824 2024-09-13T11:42:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 1 -- 172.21.15.5:0/455292394 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f281382f0 msgr2=0x7f1f28134870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 1 --2- 172.21.15.5:0/455292394 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f281382f0 0x7f1f28134870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f27fff640 1 --2- 172.21.15.5:0/455292394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f28137ac0 0x7f1f281cf7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:42:07.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 1 -- 172.21.15.5:0/455292394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f28137ac0 msgr2=0x7f1f281cf7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T11:42:07.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 1 --2- 172.21.15.5:0/455292394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f28137ac0 0x7f1f281cf7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 10 monclient: _finish_hunting 0 2024-09-13T11:42:07.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 1 monclient: found mon.noname-a 2024-09-13T11:42:07.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:07.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:07.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 1 -- 172.21.15.5:0/455292394 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f2813e120 con 0x7f1f28137ea0 2024-09-13T11:42:07.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f1ffff640 1 --2- 172.21.15.5:0/455292394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f28137ea0 0x7f1f2811ea10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800ea00 tx=0x7f1f1800eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 1 -- 172.21.15.5:0/455292394 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f18017070 con 0x7f1f28137ea0 2024-09-13T11:42:07.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 10 monclient: dump: 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 10 monclient: _finish_auth 0 2024-09-13T11:42:07.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 10 monclient: _check_auth_tickets 2024-09-13T11:42:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 1 -- 172.21.15.5:0/455292394 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f1800ca60 con 0x7f1f28137ea0 2024-09-13T11:42:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 1 -- 172.21.15.5:0/455292394 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1800ce10 con 0x7f1f28137ea0 2024-09-13T11:42:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 10 monclient: dump: 2024-09-13T11:42:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.015 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.015 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.015 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.015 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.015 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.016 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.016 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f26ffd640 4 set_mon_vals no callback set 2024-09-13T11:42:07.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 5 monclient: authenticate success, global_id 4551 2024-09-13T11:42:07.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient: _renew_subs 2024-09-13T11:42:07.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:07.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/455292394 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f280a6250 con 0x7f1f28137ea0 2024-09-13T11:42:07.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient: _renew_subs 2024-09-13T11:42:07.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:07.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/455292394 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f28134db0 con 0x7f1f28137ea0 2024-09-13T11:42:07.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:07.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:07.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:07.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/455292394 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f2809cc20 con 0x7f1f28137ea0 2024-09-13T11:42:07.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 1 -- 172.21.15.5:0/455292394 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f1f1801a070 con 0x7f1f28137ea0 2024-09-13T11:42:07.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 1 --2- 172.21.15.5:0/455292394 >> v2:172.21.15.39:6812/1340838781 conn(0x7f1f0803b2d0 0x7f1f0803d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f257fa640 1 -- 172.21.15.5:0/455292394 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f1f1804ec50 con 0x7f1f28137ea0 2024-09-13T11:42:07.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f27fff640 1 --2- 172.21.15.5:0/455292394 >> v2:172.21.15.39:6812/1340838781 conn(0x7f1f0803b2d0 0x7f1f0803d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.983+0000 7f1f27fff640 10 monclient: get_auth_request con 0x7f1f0803b2d0 auth_method 0 2024-09-13T11:42:07.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.987+0000 7f1f27fff640 1 --2- 172.21.15.5:0/455292394 >> v2:172.21.15.39:6812/1340838781 conn(0x7f1f0803b2d0 0x7f1f0803d790 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000b990 tx=0x7f1f10008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.991+0000 7f1f257fa640 1 -- 172.21.15.5:0/455292394 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f1f1804da80 con 0x7f1f28137ea0 2024-09-13T11:42:07.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.991+0000 7f1f257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:07.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:06.991+0000 7f1f257fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:07.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:42:07.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:42:07.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:07.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/455292394 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1f2809e0b0 con 0x7f1f28137ea0 2024-09-13T11:42:07.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f257fa640 1 -- 172.21.15.5:0/455292394 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1548 (secure 0 0 0) 0x7f1f2809e0b0 con 0x7f1f28137ea0 2024-09-13T11:42:07.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:42:07.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f257fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:42:07.230 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","modified":"2024-09-13T11:39:20.750174Z","created":"2024-09-13T11:39:20.750174Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:42:07.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/455292394 >> v2:172.21.15.39:6812/1340838781 conn(0x7f1f0803b2d0 msgr2=0x7f1f0803d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 --2- 172.21.15.5:0/455292394 >> v2:172.21.15.39:6812/1340838781 conn(0x7f1f0803b2d0 0x7f1f0803d790 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000b990 tx=0x7f1f10008040 comp rx=0 tx=0).stop 2024-09-13T11:42:07.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 10 monclient: shutdown 2024-09-13T11:42:07.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:07.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/455292394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f28137ea0 msgr2=0x7f1f2811ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 --2- 172.21.15.5:0/455292394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f28137ea0 0x7f1f2811ea10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800ea00 tx=0x7f1f1800eec0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/455292394 shutdown_connections 2024-09-13T11:42:07.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 --2- 172.21.15.5:0/455292394 >> v2:172.21.15.39:6812/1340838781 conn(0x7f1f0803b2d0 0x7f1f0803d790 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 --2- 172.21.15.5:0/455292394 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f281382f0 0x7f1f28134870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 --2- 172.21.15.5:0/455292394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f28137ea0 0x7f1f2811ea10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 --2- 172.21.15.5:0/455292394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f28137ac0 0x7f1f281cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/455292394 >> 172.21.15.5:0/455292394 conn(0x7f1f280a39f0 msgr2=0x7f1f2809f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:07.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/455292394 shutdown_connections 2024-09-13T11:42:07.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.227+0000 7f1f2e4c2640 1 -- 172.21.15.5:0/455292394 wait complete. 2024-09-13T11:42:07.247 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T11:42:07.247 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T11:42:07.248 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T11:42:07.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 10 monclient: get_monmap_and_config 2024-09-13T11:42:07.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 10 monclient: build_initial_monmap 2024-09-13T11:42:07.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 10 monclient: monmap: 2024-09-13T11:42:07.358 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:07.358 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.358 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:07.357884+0000 2024-09-13T11:42:07.358 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:07.357884+0000 2024-09-13T11:42:07.358 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:07.358 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.358 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:07.358 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:07.358 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:07.359 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:07.359 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:07.359 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 1 Processor -- start 2024-09-13T11:42:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 1 -- start start 2024-09-13T11:42:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 10 monclient: init 2024-09-13T11:42:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-13T11:42:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7dc134810 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7dc137660 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb7dc13b170 0x7fb7dc13d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 10 monclient(hunting): picked mon.noname-c con 0x7fb7dc13b170 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7dc09ce60 con 0x7fb7dc134810 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7dc09eeb0 con 0x7fb7dc137660 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fb7dc0a0950 con 0x7fb7dc13b170 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 10 monclient(hunting): authenticate will time out at 1130.706572s 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fb7dc134810 0x7fb7dc134bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:56922/0 (socket says 172.21.15.5:56922) 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 1 -- 172.21.15.5:0/4217757122 learned_addr learned my addr 172.21.15.5:0/4217757122 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e2412640 1 --2- 172.21.15.5:0/4217757122 >> v2:172.21.15.39:3300/0 conn(0x7fb7dc13b170 0x7fb7dc13d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e2412640 10 monclient(hunting): get_auth_request con 0x7fb7dc13b170 auth_method 0 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e2412640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e2412640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e2412640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3035844366 0 0) 0x7fb7dc09ce60 con 0x7fb7dc134810 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient(hunting): _negotiate 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient(hunting): my global_id is 4554 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7c8003530 con 0x7fb7dc134810 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e2412640 1 -- 172.21.15.5:0/4217757122 >> v2:172.21.15.39:3300/0 conn(0x7fb7dc13b170 msgr2=0x7fb7dc13d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e2412640 1 -- 172.21.15.5:0/4217757122 >> v2:172.21.15.39:3300/0 conn(0x7fb7dc13b170 msgr2=0x7fb7dc13d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e2412640 1 --2- 172.21.15.5:0/4217757122 >> v2:172.21.15.39:3300/0 conn(0x7fb7dc13b170 0x7fb7dc13d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e2412640 1 --2- 172.21.15.5:0/4217757122 >> v2:172.21.15.39:3300/0 conn(0x7fb7dc13b170 0x7fb7dc13d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1446696541 0 0) 0x7fb7c8003530 con 0x7fb7dc134810 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 >> v2:172.21.15.39:3300/0 conn(0x7fb7dc13b170 msgr2=0x7fb7dc13d560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 --2- 172.21.15.5:0/4217757122 >> v2:172.21.15.39:3300/0 conn(0x7fb7dc13b170 0x7fb7dc13d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 >> v1:172.21.15.39:6789/0 conn(0x7fb7dc137660 legacy=0x7fb7dc139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient: _finish_hunting 0 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7dc09e1e0 con 0x7fb7dc134810 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient: _finish_auth 0 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient: _check_auth_tickets 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 5 monclient: authenticate success, global_id 4554 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3199362888 0 0) 0x7fb7cc001700 con 0x7fb7dc134810 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:42:07.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient: dump: 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 >> v1:172.21.15.5:6789/0 conn(0x7fb7dc134810 legacy=0x7fb7dc134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:42:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 --2- 172.21.15.5:0/4217757122 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7c8003ab0 0x7fb7c8003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient(hunting): picked mon.a con 0x7fb7c8003ab0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 --2- 172.21.15.5:0/4217757122 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7c8004810 0x7fb7c8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient(hunting): picked mon.c con 0x7fb7c8004810 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 --2- 172.21.15.5:0/4217757122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7c8005400 0x7fb7c800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient(hunting): picked mon.b con 0x7fb7c8005400 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c800c320 con 0x7fb7c8003ab0 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c800c4d0 con 0x7fb7c8005400 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fb7c800c6d0 con 0x7fb7c8004810 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e0c0f640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e3e75640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 1 --2- 172.21.15.5:0/4217757122 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7c8003ab0 0x7fb7c8003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1410640 1 --2- 172.21.15.5:0/4217757122 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7c8004810 0x7fb7c8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 10 monclient(hunting): get_auth_request con 0x7fb7c8003ab0 auth_method 0 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1410640 10 monclient(hunting): get_auth_request con 0x7fb7c8004810 auth_method 0 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1410640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1410640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1410640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1410640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1410640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1410640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e2412640 1 --2- 172.21.15.5:0/4217757122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7c8005400 0x7fb7c800ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_done global_id 4554 payload 293 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 1 -- 172.21.15.5:0/4217757122 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7c8004810 msgr2=0x7fb7c8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 1 --2- 172.21.15.5:0/4217757122 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7c8004810 0x7fb7c8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 1 -- 172.21.15.5:0/4217757122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7c8005400 msgr2=0x7fb7c800ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 1 --2- 172.21.15.5:0/4217757122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7c8005400 0x7fb7c800ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 10 monclient: _finish_hunting 0 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 1 monclient: found mon.a 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 1 -- 172.21.15.5:0/4217757122 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c8003530 con 0x7fb7c8003ab0 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 10 monclient: _finish_auth 0 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 10 monclient: _check_auth_tickets 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e1c11640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.355+0000 7fb7e2412640 10 monclient: get_auth_request con 0x7fb7c8005400 auth_method 0 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e2412640 1 --2- 172.21.15.5:0/4217757122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7c8005400 0x7fb7c800ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e2412640 1 -- 172.21.15.5:0/4217757122 reap_dead start 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e1c11640 1 --2- 172.21.15.5:0/4217757122 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7c8003ab0 0x7fb7c8003ed0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00c190 tx=0x7fb7cc01cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6fd58bc49a78901 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc007c90 con 0x7fb7c8003ab0 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e0c0f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e0c0f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e0c0f640 10 monclient: dump: 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7cc0a9020 con 0x7fb7c8003ab0 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e0c0f640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e0c0f640 1 -- 172.21.15.5:0/4217757122 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc005b20 con 0x7fb7c8003ab0 2024-09-13T11:42:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 4 set_mon_vals no callback set 2024-09-13T11:42:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e0c0f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e0c0f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e0c0f640 10 monclient: dump: 2024-09-13T11:42:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 10 monclient: shutdown 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 1 -- 172.21.15.5:0/4217757122 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7c8003ab0 msgr2=0x7fb7c8003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 1 --2- 172.21.15.5:0/4217757122 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7c8003ab0 0x7fb7c8003ed0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00c190 tx=0x7fb7cc01cb00 comp rx=0 tx=0).stop 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 1 -- 172.21.15.5:0/4217757122 shutdown_connections 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 1 --2- 172.21.15.5:0/4217757122 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7c8003ab0 0x7fb7c8003ed0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 1 -- 172.21.15.5:0/4217757122 >> 172.21.15.5:0/4217757122 conn(0x7fb7dc0a2930 msgr2=0x7fb7dc0a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 1 -- 172.21.15.5:0/4217757122 shutdown_connections 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 1 -- 172.21.15.5:0/4217757122 wait complete. 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 10 monclient: build_initial_monmap 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.359+0000 7fb7e3e75640 10 monclient: monmap: 2024-09-13T11:42:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 1 Processor -- start 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 1 -- start start 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient: init 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7c8003ab0 0x7fb7dc1332e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient(hunting): picked mon.noname-a con 0x7fb7c8003ab0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7dc133820 0x7fb7dc11ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient(hunting): picked mon.noname-c con 0x7fb7dc133820 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7dc11d3c0 0x7fb7dc11d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient(hunting): picked mon.noname-b con 0x7fb7dc11d3c0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb7dc09d560 con 0x7fb7c8003ab0 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb7dc09d3e0 con 0x7fb7dc11d3c0 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fb7dc09d6e0 con 0x7fb7dc133820 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient(hunting): authenticate will time out at 1130.712785s 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7c8003ab0 0x7fb7dc1332e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7dc133820 0x7fb7dc11ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7c8003ab0 0x7fb7dc1332e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52572/0 (socket says 172.21.15.5:52572) 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 1 -- 172.21.15.5:0/1809018975 learned_addr learned my addr 172.21.15.5:0/1809018975 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): get_auth_request con 0x7fb7c8003ab0 auth_method 0 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 10 monclient(hunting): get_auth_request con 0x7fb7dc133820 auth_method 0 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e2412640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7dc11d3c0 0x7fb7dc11d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 10 monclient(hunting): handle_auth_done global_id 4409 payload 824 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 1 -- 172.21.15.5:0/1809018975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7dc11d3c0 msgr2=0x7fb7dc11d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7dc11d3c0 0x7fb7dc11d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 1 -- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7c8003ab0 msgr2=0x7fb7dc1332e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7c8003ab0 0x7fb7dc1332e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 10 monclient: _finish_hunting 0 2024-09-13T11:42:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 1 monclient: found mon.noname-c 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 1 -- 172.21.15.5:0/1809018975 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7dc11e940 con 0x7fb7dc133820 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7c8003ab0 0x7fb7dc1332e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e2412640 10 monclient: get_auth_request con 0x7fb7dc11d3c0 auth_method 0 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e2412640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7dc11d3c0 0x7fb7dc11d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1410640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7dc133820 0x7fb7dc11ce80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc06a880 tx=0x7fb7c400ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4019070 con 0x7fb7dc133820 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: dump: 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: _finish_auth 0 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7c4006d70 con 0x7fb7dc133820 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4003bc0 con 0x7fb7dc133820 2024-09-13T11:42:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 5 monclient: authenticate success, global_id 4409 2024-09-13T11:42:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient: _renew_subs 2024-09-13T11:42:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 1 -- 172.21.15.5:0/1809018975 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7dc133ca0 con 0x7fb7dc133820 2024-09-13T11:42:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7dbfff640 4 set_mon_vals no callback set 2024-09-13T11:42:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: dump: 2024-09-13T11:42:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient: _renew_subs 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 1 -- 172.21.15.5:0/1809018975 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7dc1206e0 con 0x7fb7dc133820 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc02e380 0x7fb7dc11dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e3e75640 1 -- 172.21.15.5:0/1809018975 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb7dc02e760 con 0x7fb7dc02e380 2024-09-13T11:42:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fb7c401c070 con 0x7fb7dc133820 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc02e380 0x7fb7dc11dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient: get_auth_request con 0x7fb7dc02e380 auth_method 0 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 1 --2- 172.21.15.5:0/1809018975 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb7bc03b2a0 0x7fb7bc03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7fb7c404e360 con 0x7fb7dc133820 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_done global_id 4560 payload 824 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 1 -- 172.21.15.5:0/1809018975 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7dc11e2b0 con 0x7fb7dc02e380 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e1c11640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc02e380 0x7fb7dc11dd70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc01b200 tx=0x7fb7cc019940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e2412640 1 --2- 172.21.15.5:0/1809018975 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb7bc03b2a0 0x7fb7bc03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc001700 con 0x7fb7dc02e380 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: dump: 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.393 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e2412640 10 monclient: get_auth_request con 0x7fb7bc03b2a0 auth_method 0 2024-09-13T11:42:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb7cc0a9030 con 0x7fb7dc02e380 2024-09-13T11:42:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc02e380 msgr2=0x7fb7dc11dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc02e380 0x7fb7dc11dd70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc01b200 tx=0x7fb7cc019940 comp rx=0 tx=0).stop 2024-09-13T11:42:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc02e380 msgr2=0x7fb7dc11dd70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:42:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7da7fc640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc02e380 0x7fb7dc11dd70 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc01b200 tx=0x7fb7cc019940 comp rx=0 tx=0).stop 2024-09-13T11:42:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.363+0000 7fb7e2412640 1 --2- 172.21.15.5:0/1809018975 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb7bc03b2a0 0x7fb7bc03d760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc06e020 tx=0x7fb7d4006c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:42:07.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc063b60 0x7fb7dc063f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 1 -- 172.21.15.5:0/1809018975 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb7dc09da70 con 0x7fb7dc063b60 2024-09-13T11:42:07.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e1c11640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc063b60 0x7fb7dc063f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e1c11640 10 monclient: get_auth_request con 0x7fb7dc063b60 auth_method 0 2024-09-13T11:42:07.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e1c11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e1c11640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e1c11640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e1c11640 10 monclient(hunting): handle_auth_done global_id 4563 payload 824 2024-09-13T11:42:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e1c11640 1 -- 172.21.15.5:0/1809018975 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb7dc071e40 con 0x7fb7dc063b60 2024-09-13T11:42:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e1c11640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc063b60 0x7fb7dc063f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc02e8e0 tx=0x7fb7c8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0c4070 con 0x7fb7dc063b60 2024-09-13T11:42:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7da7fc640 10 monclient: dump: 2024-09-13T11:42:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.397 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7fb7cc0be2b0 con 0x7fb7dc063b60 2024-09-13T11:42:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7da7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:42:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:42:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc063b60 msgr2=0x7fb7dc063f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7da7fc640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc063b60 0x7fb7dc063f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc02e8e0 tx=0x7fb7c8005840 comp rx=0 tx=0).stop 2024-09-13T11:42:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7da7fc640 1 -- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc063b60 msgr2=0x7fb7dc063f70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:42:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7da7fc640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7dc063b60 0x7fb7dc063f70 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc02e8e0 tx=0x7fb7c8005840 comp rx=0 tx=0).stop 2024-09-13T11:42:07.398 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:42:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-13T11:42:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-13T11:42:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 18, 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "uptime": 139343, 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2450521708007866372", 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:42:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:42:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-13T11:42:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "59046471-79a9-4d90-9616-da0b446053a7", 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 19, 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "squid", 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 1, 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-13T11:42:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-13T11:42:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:3300", 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789", 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789/0", 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.39:6789/0", 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:42:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:42:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:42:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:42:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:42:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-13T11:42:07.411 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:42:07.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 1 -- 172.21.15.5:0/1809018975 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb7bc03b2a0 msgr2=0x7fb7bc03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 1 --2- 172.21.15.5:0/1809018975 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb7bc03b2a0 0x7fb7bc03d760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc06e020 tx=0x7fb7d4006c00 comp rx=0 tx=0).stop 2024-09-13T11:42:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 10 monclient: shutdown 2024-09-13T11:42:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 1 -- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7dc133820 msgr2=0x7fb7dc11ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 1 --2- 172.21.15.5:0/1809018975 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb7dc133820 0x7fb7dc11ce80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc06a880 tx=0x7fb7c400ff70 comp rx=0 tx=0).stop 2024-09-13T11:42:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e2412640 1 -- 172.21.15.5:0/1809018975 reap_dead start 2024-09-13T11:42:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e2412640 1 -- 172.21.15.5:0/1809018975 reap_dead start 2024-09-13T11:42:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 1 -- 172.21.15.5:0/1809018975 shutdown_connections 2024-09-13T11:42:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 1 -- 172.21.15.5:0/1809018975 >> 172.21.15.5:0/1809018975 conn(0x7fb7dc0a2930 msgr2=0x7fb7dc06f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 1 -- 172.21.15.5:0/1809018975 shutdown_connections 2024-09-13T11:42:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.371+0000 7fb7e3e75640 1 -- 172.21.15.5:0/1809018975 wait complete. 2024-09-13T11:42:07.415 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:42:07.415 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T11:42:07.415 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T11:42:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 10 monclient: get_monmap_and_config 2024-09-13T11:42:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 10 monclient: build_initial_monmap 2024-09-13T11:42:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 10 monclient: monmap: 2024-09-13T11:42:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:07.525056+0000 2024-09-13T11:42:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:07.525056+0000 2024-09-13T11:42:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:07.526 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 1 Processor -- start 2024-09-13T11:42:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 1 -- start start 2024-09-13T11:42:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 10 monclient: init 2024-09-13T11:42:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-13T11:42:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faaa0135820 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0073da0 0x7faaa013a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 10 monclient(hunting): picked mon.noname-b con 0x7faaa0073da0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faaa013ac70 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa00741e0 con 0x7faaa0135820 2024-09-13T11:42:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa01353d0 con 0x7faaa013ac70 2024-09-13T11:42:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7faaa00a0f70 con 0x7faaa0073da0 2024-09-13T11:42:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa8369640 10 monclient(hunting): authenticate will time out at 1130.873983s 2024-09-13T11:42:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0073da0 0x7faaa013a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa6105640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7faaa0135820 0x7faaa0135c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:56938/0 (socket says 172.21.15.5:56938) 2024-09-13T11:42:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa6105640 1 -- 172.21.15.5:0/3784659009 learned_addr learned my addr 172.21.15.5:0/3784659009 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:42:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 10 monclient(hunting): get_auth_request con 0x7faaa0073da0 auth_method 0 2024-09-13T11:42:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5103640 1 -- 172.21.15.5:0/3784659009 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 295768973 0 0) 0x7faa980014a0 con 0x7faaa0135820 2024-09-13T11:42:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5103640 10 monclient(hunting): _negotiate 2024-09-13T11:42:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5103640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5103640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5103640 10 monclient(hunting): my global_id is 4566 2024-09-13T11:42:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5103640 1 -- 172.21.15.5:0/3784659009 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa88003530 con 0x7faaa0135820 2024-09-13T11:42:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 10 monclient(hunting): handle_auth_done global_id 4412 payload 274 2024-09-13T11:42:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 1 -- 172.21.15.5:0/3784659009 >> v1:172.21.15.39:6789/0 conn(0x7faaa013ac70 legacy=0x7faaa013d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 1 -- 172.21.15.5:0/3784659009 >> v1:172.21.15.5:6789/0 conn(0x7faaa0135820 legacy=0x7faaa0135c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 10 monclient: _finish_hunting 0 2024-09-13T11:42:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 1 monclient: found mon.noname-b 2024-09-13T11:42:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.523+0000 7faaa5904640 1 -- 172.21.15.5:0/3784659009 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa0072af0 con 0x7faaa0073da0 2024-09-13T11:42:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5904640 1 --2- 172.21.15.5:0/3784659009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0073da0 0x7faaa013a730 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faa940097f0 tx=0x7faa9401c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=311adabd6fb2e8f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 1 -- 172.21.15.5:0/3784659009 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa940b0070 con 0x7faaa0073da0 2024-09-13T11:42:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:42:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 10 monclient: dump: 2024-09-13T11:42:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 10 monclient: _finish_auth 0 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 10 monclient: _check_auth_tickets 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 5 monclient: authenticate success, global_id 4412 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 1 -- 172.21.15.5:0/3784659009 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa940ab600 con 0x7faaa0073da0 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 4 set_mon_vals no callback set 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 1 -- 172.21.15.5:0/3784659009 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa940ab9d0 con 0x7faaa0073da0 2024-09-13T11:42:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5103640 10 monclient: dump: 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient: shutdown 2024-09-13T11:42:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 -- 172.21.15.5:0/3784659009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0073da0 msgr2=0x7faaa013a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 --2- 172.21.15.5:0/3784659009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0073da0 0x7faaa013a730 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faa940097f0 tx=0x7faa9401c430 comp rx=0 tx=0).stop 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 -- 172.21.15.5:0/3784659009 shutdown_connections 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 --2- 172.21.15.5:0/3784659009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0073da0 0x7faaa013a730 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 -- 172.21.15.5:0/3784659009 >> 172.21.15.5:0/3784659009 conn(0x7faaa00a3710 msgr2=0x7faaa00a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 -- 172.21.15.5:0/3784659009 shutdown_connections 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 -- 172.21.15.5:0/3784659009 wait complete. 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient: build_initial_monmap 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient: monmap: 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 Processor -- start 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 -- start start 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient: init 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0073da0 0x7faaa012b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient(hunting): picked mon.noname-c con 0x7faaa0073da0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faaa0135820 0x7faaa012b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient(hunting): picked mon.noname-b con 0x7faaa0135820 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaa013ac70 0x7faaa01373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient(hunting): picked mon.noname-a con 0x7faaa013ac70 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7faaa01411b0 con 0x7faaa013ac70 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7faaa00763a0 con 0x7faaa0135820 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7faaa0141330 con 0x7faaa0073da0 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient(hunting): authenticate will time out at 1130.878766s 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0073da0 0x7faaa012b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0073da0 0x7faaa012b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:48450/0 (socket says 172.21.15.5:48450) 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 1 -- 172.21.15.5:0/1726780873 learned_addr learned my addr 172.21.15.5:0/1726780873 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6906640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaa013ac70 0x7faaa01373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 10 monclient(hunting): get_auth_request con 0x7faaa0073da0 auth_method 0 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6906640 10 monclient(hunting): get_auth_request con 0x7faaa013ac70 auth_method 0 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6906640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6906640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6906640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6906640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6906640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5904640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faaa0135820 0x7faaa012b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 10 monclient(hunting): handle_auth_done global_id 4415 payload 824 2024-09-13T11:42:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 1 -- 172.21.15.5:0/1726780873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faaa0135820 msgr2=0x7faaa012b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faaa0135820 0x7faaa012b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 1 -- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaa013ac70 msgr2=0x7faaa01373d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaa013ac70 0x7faaa01373d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 10 monclient: _finish_hunting 0 2024-09-13T11:42:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 1 monclient: found mon.noname-c 2024-09-13T11:42:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 1 -- 172.21.15.5:0/1726780873 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa013a520 con 0x7faaa0073da0 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5904640 10 monclient: get_auth_request con 0x7faaa0135820 auth_method 0 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa5904640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faaa0135820 0x7faaa012b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa6105640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0073da0 0x7faaa012b1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faa9800cbc0 tx=0x7faa9800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9800ad40 con 0x7faaa0073da0 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 10 monclient: dump: 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 10 monclient: _finish_auth 0 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 10 monclient: _check_auth_tickets 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa98012400 con 0x7faaa0073da0 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa980127f0 con 0x7faaa0073da0 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9effd640 10 monclient: dump: 2024-09-13T11:42:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.553 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 5 monclient: authenticate success, global_id 4415 2024-09-13T11:42:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient: _renew_subs 2024-09-13T11:42:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 -- 172.21.15.5:0/1726780873 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaa0074a10 con 0x7faaa0073da0 2024-09-13T11:42:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faa9ffff640 4 set_mon_vals no callback set 2024-09-13T11:42:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient: _renew_subs 2024-09-13T11:42:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.527+0000 7faaa8369640 1 -- 172.21.15.5:0/1726780873 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faaa0139970 con 0x7faaa0073da0 2024-09-13T11:42:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa8369640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa8369640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0137910 0x7faaa0137cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa8369640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa8369640 1 -- 172.21.15.5:0/1726780873 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7faaa00a5a90 con 0x7faaa0137910 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa5904640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0137910 0x7faaa0137cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa5904640 10 monclient: get_auth_request con 0x7faaa0137910 auth_method 0 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa5904640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa5904640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa5904640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa5904640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa5904640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa5904640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7faa98012990 con 0x7faaa0073da0 2024-09-13T11:42:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa5904640 10 monclient(hunting): handle_auth_done global_id 4418 payload 824 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa5904640 1 -- 172.21.15.5:0/1726780873 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faaa012fbc0 con 0x7faaa0137910 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 1 --2- 172.21.15.5:0/1726780873 >> v2:172.21.15.39:6812/1340838781 conn(0x7faa8c03b2a0 0x7faa8c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7faa9804f2c0 con 0x7faaa0073da0 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa5904640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0137910 0x7faaa0137cf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faaa006e480 tx=0x7faa9401ac70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa940bb070 con 0x7faaa0137910 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 10 monclient: dump: 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.559 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa6906640 1 --2- 172.21.15.5:0/1726780873 >> v2:172.21.15.39:6812/1340838781 conn(0x7faa8c03b2a0 0x7faa8c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa6906640 10 monclient: get_auth_request con 0x7faa8c03b2a0 auth_method 0 2024-09-13T11:42:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faa940b1040 con 0x7faaa0137910 2024-09-13T11:42:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0137910 msgr2=0x7faaa0137cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0137910 0x7faaa0137cf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faaa006e480 tx=0x7faa9401ac70 comp rx=0 tx=0).stop 2024-09-13T11:42:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0137910 msgr2=0x7faaa0137cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:42:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faa9effd640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0137910 0x7faaa0137cf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.531+0000 7faaa6906640 1 --2- 172.21.15.5:0/1726780873 >> v2:172.21.15.39:6812/1340838781 conn(0x7faa8c03b2a0 0x7faa8c03d760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faaa0139700 tx=0x7faa90009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa8369640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:42:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa8369640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0138230 0x7faaa013b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa8369640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa8369640 1 -- 172.21.15.5:0/1726780873 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7faaa0069ff0 con 0x7faaa0138230 2024-09-13T11:42:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa5904640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0138230 0x7faaa013b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa5904640 10 monclient: get_auth_request con 0x7faaa0138230 auth_method 0 2024-09-13T11:42:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa5904640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa5904640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa5904640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa5904640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa5904640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa5904640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa5904640 10 monclient(hunting): handle_auth_done global_id 4421 payload 824 2024-09-13T11:42:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa5904640 1 -- 172.21.15.5:0/1726780873 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faaa012f5e0 con 0x7faaa0138230 2024-09-13T11:42:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faaa5904640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0138230 0x7faaa013b0b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faa940097f0 tx=0x7faa940191d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa940bb050 con 0x7faaa0138230 2024-09-13T11:42:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faa9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faa9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.535+0000 7faa9effd640 10 monclient: dump: 2024-09-13T11:42:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.564 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7faa940bc040 con 0x7faaa0138230 2024-09-13T11:42:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faa9effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:42:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faa9effd640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:42:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0138230 msgr2=0x7faaa013b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faa9effd640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0138230 0x7faaa013b0b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faa940097f0 tx=0x7faa940191d0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faa9effd640 1 -- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0138230 msgr2=0x7faaa013b0b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:42:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faa9effd640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0138230 0x7faaa013b0b0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faa940097f0 tx=0x7faa940191d0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.565 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:42:07.565 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-13T11:42:07.565 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-13T11:42:07.565 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-13T11:42:07.566 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 18, 2024-09-13T11:42:07.566 INFO:teuthology.orchestra.run.smithi005.stdout: "uptime": 139507, 2024-09-13T11:42:07.566 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-13T11:42:07.566 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-13T11:42:07.566 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-13T11:42:07.566 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-13T11:42:07.566 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.566 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2450521708007866372", 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:42:07.567 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:42:07.568 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.568 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:42:07.568 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-13T11:42:07.568 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:42:07.568 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:42:07.568 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:42:07.568 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:42:07.568 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:42:07.568 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:42:07.568 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:42:07.568 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:42:07.569 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:42:07.569 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:42:07.569 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:42:07.569 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.569 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-13T11:42:07.569 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-13T11:42:07.569 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-13T11:42:07.569 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-13T11:42:07.569 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-13T11:42:07.569 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "59046471-79a9-4d90-9616-da0b446053a7", 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 19, 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "squid", 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 1, 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:42:07.570 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-13T11:42:07.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.572 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-13T11:42:07.572 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-13T11:42:07.572 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:42:07.572 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:42:07.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.572 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:42:07.572 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-13T11:42:07.572 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.572 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:42:07.572 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-13T11:42:07.573 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.573 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.573 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.573 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-13T11:42:07.573 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-13T11:42:07.573 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:42:07.573 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:42:07.573 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:42:07.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:3300", 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789", 2024-09-13T11:42:07.574 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789/0", 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.39:6789/0", 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:42:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-13T11:42:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-13T11:42:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:42:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:42:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:42:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-13T11:42:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-13T11:42:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:42:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:42:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:42:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-13T11:42:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:42:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:42:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:42:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-13T11:42:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:42:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:42:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-13T11:42:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-13T11:42:07.580 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:42:07.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faaa8369640 1 -- 172.21.15.5:0/1726780873 >> v2:172.21.15.39:6812/1340838781 conn(0x7faa8c03b2a0 msgr2=0x7faa8c03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faaa8369640 1 --2- 172.21.15.5:0/1726780873 >> v2:172.21.15.39:6812/1340838781 conn(0x7faa8c03b2a0 0x7faa8c03d760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faaa0139700 tx=0x7faa90009040 comp rx=0 tx=0).stop 2024-09-13T11:42:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faaa8369640 10 monclient: shutdown 2024-09-13T11:42:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faaa8369640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faaa8369640 1 -- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0073da0 msgr2=0x7faaa012b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faaa8369640 1 --2- 172.21.15.5:0/1726780873 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faaa0073da0 0x7faaa012b1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faa9800cbc0 tx=0x7faa9800a430 comp rx=0 tx=0).stop 2024-09-13T11:42:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faaa6906640 1 -- 172.21.15.5:0/1726780873 reap_dead start 2024-09-13T11:42:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faaa6906640 1 -- 172.21.15.5:0/1726780873 reap_dead start 2024-09-13T11:42:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faaa8369640 1 -- 172.21.15.5:0/1726780873 shutdown_connections 2024-09-13T11:42:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faaa8369640 1 -- 172.21.15.5:0/1726780873 >> 172.21.15.5:0/1726780873 conn(0x7faaa00a3710 msgr2=0x7faaa0133d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faaa8369640 1 -- 172.21.15.5:0/1726780873 shutdown_connections 2024-09-13T11:42:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.539+0000 7faaa8369640 1 -- 172.21.15.5:0/1726780873 wait complete. 2024-09-13T11:42:07.583 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:42:07.583 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T11:42:07.583 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T11:42:07.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient: get_monmap_and_config 2024-09-13T11:42:07.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient: build_initial_monmap 2024-09-13T11:42:07.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient: monmap: 2024-09-13T11:42:07.693 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:07.693 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.694 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:07.693485+0000 2024-09-13T11:42:07.694 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:07.693485+0000 2024-09-13T11:42:07.694 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:07.694 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.694 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:07.694 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:07.694 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:07.694 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:07.694 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:07.695 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 1 Processor -- start 2024-09-13T11:42:07.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 1 -- start start 2024-09-13T11:42:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient: init 2024-09-13T11:42:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-13T11:42:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f54f0137930 0x7f54f009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient(hunting): picked mon.noname-c con 0x7f54f0137930 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54f0137d10 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f54f013cd80 0x7f54f013f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient(hunting): picked mon.noname-a con 0x7f54f013cd80 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54f00a5a80 con 0x7f54f0137d10 2024-09-13T11:42:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f54f00a5fe0 con 0x7f54f013cd80 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f54f009c540 con 0x7f54f0137930 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f6aca640 10 monclient(hunting): authenticate will time out at 1131.042320s 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54effff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f54f0137d10 0x7f54f01380f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60670/0 (socket says 172.21.15.5:60670) 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54effff640 1 -- 172.21.15.5:0/1053005924 learned_addr learned my addr 172.21.15.5:0/1053005924 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 1 --2- 172.21.15.5:0/1053005924 >> v2:172.21.15.5:3300/0 conn(0x7f54f013cd80 0x7f54f013f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 10 monclient(hunting): get_auth_request con 0x7f54f013cd80 auth_method 0 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f4866640 1 --2- 172.21.15.5:0/1053005924 >> v2:172.21.15.39:3300/0 conn(0x7f54f0137930 0x7f54f009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f4866640 10 monclient(hunting): get_auth_request con 0x7f54f0137930 auth_method 0 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f4866640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f4866640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f4866640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 10 monclient(hunting): handle_auth_done global_id 4572 payload 274 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 1 -- 172.21.15.5:0/1053005924 >> v2:172.21.15.39:3300/0 conn(0x7f54f0137930 msgr2=0x7f54f009cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 1 --2- 172.21.15.5:0/1053005924 >> v2:172.21.15.39:3300/0 conn(0x7f54f0137930 0x7f54f009cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 1 -- 172.21.15.5:0/1053005924 >> v1:172.21.15.5:6789/0 conn(0x7f54f0137d10 legacy=0x7f54f01380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 10 monclient: _finish_hunting 0 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 1 monclient: found mon.noname-a 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:07.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.691+0000 7f54f5067640 1 -- 172.21.15.5:0/1053005924 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f009bef0 con 0x7f54f013cd80 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f5067640 1 --2- 172.21.15.5:0/1053005924 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54f013cd80 0x7f54f013f170 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f54f00a7cb0 tx=0x7f54e801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107504e5a3552d5f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 1 -- 172.21.15.5:0/1053005924 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e80b2070 con 0x7f54f013cd80 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 10 monclient: dump: 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 10 monclient: _finish_auth 0 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 5 monclient: authenticate success, global_id 4572 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 1 -- 172.21.15.5:0/1053005924 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54e80a57c0 con 0x7f54f013cd80 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 1 -- 172.21.15.5:0/1053005924 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e80a5b90 con 0x7f54f013cd80 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54ef7fe640 10 monclient: dump: 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 4 set_mon_vals no callback set 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 10 monclient: shutdown 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 1 -- 172.21.15.5:0/1053005924 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54f013cd80 msgr2=0x7f54f013f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 1 --2- 172.21.15.5:0/1053005924 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54f013cd80 0x7f54f013f170 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f54f00a7cb0 tx=0x7f54e801c7f0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 1 -- 172.21.15.5:0/1053005924 shutdown_connections 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 1 --2- 172.21.15.5:0/1053005924 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54f013cd80 0x7f54f013f170 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 1 --2- 172.21.15.5:0/1053005924 >> v2:172.21.15.39:3300/0 conn(0x7f54f0137930 0x7f54f009cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 1 -- 172.21.15.5:0/1053005924 >> 172.21.15.5:0/1053005924 conn(0x7f54f00a6c70 msgr2=0x7f54f00a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 1 -- 172.21.15.5:0/1053005924 shutdown_connections 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 1 -- 172.21.15.5:0/1053005924 wait complete. 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 10 monclient: build_initial_monmap 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 10 monclient: monmap: 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 1 Processor -- start 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 1 -- start start 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 10 monclient: init 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54f0137930 0x7f54f0136400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 10 monclient(hunting): picked mon.noname-c con 0x7f54f0137930 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f0137d10 0x7f54f0136940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.695+0000 7f54f6aca640 10 monclient(hunting): picked mon.noname-b con 0x7f54f0137d10 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54f013cd80 0x7f54f0139b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 10 monclient(hunting): picked mon.noname-a con 0x7f54f013cd80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f54f0142350 con 0x7f54f013cd80 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f54f01421d0 con 0x7f54f0137d10 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f54f01424d0 con 0x7f54f0137930 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 10 monclient(hunting): authenticate will time out at 1131.048075s 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54f013cd80 0x7f54f0139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54f0137930 0x7f54f0136400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54f013cd80 0x7f54f0139b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42854/0 (socket says 172.21.15.5:42854) 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 1 -- 172.21.15.5:0/3649158973 learned_addr learned my addr 172.21.15.5:0/3649158973 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 10 monclient(hunting): get_auth_request con 0x7f54f013cd80 auth_method 0 2024-09-13T11:42:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): get_auth_request con 0x7f54f0137930 auth_method 0 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54effff640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f0137d10 0x7f54f0136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54effff640 10 monclient(hunting): get_auth_request con 0x7f54f0137d10 auth_method 0 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54effff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 10 monclient(hunting): handle_auth_done global_id 4578 payload 824 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 1 -- 172.21.15.5:0/3649158973 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54f0137930 msgr2=0x7f54f0136400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54f0137930 0x7f54f0136400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 1 -- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f0137d10 msgr2=0x7f54f0136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f0137d10 0x7f54f0136940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 10 monclient: _finish_hunting 0 2024-09-13T11:42:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 1 monclient: found mon.noname-a 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 1 -- 172.21.15.5:0/3649158973 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f0121040 con 0x7f54f013cd80 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f5067640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54f013cd80 0x7f54f0139b50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f54e8019400 tx=0x7f54e8019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e80b2050 con 0x7f54f013cd80 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 10 monclient: dump: 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 10 monclient: _finish_auth 0 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 10 monclient: _check_auth_tickets 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 5 monclient: authenticate success, global_id 4578 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54e80be070 con 0x7f54f013cd80 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 10 monclient: _renew_subs 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 1 -- 172.21.15.5:0/3649158973 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54f00a93b0 con 0x7f54f013cd80 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e80ada90 con 0x7f54f013cd80 2024-09-13T11:42:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 10 monclient: dump: 2024-09-13T11:42:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.723 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 10 monclient: _renew_subs 2024-09-13T11:42:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54eeffd640 4 set_mon_vals no callback set 2024-09-13T11:42:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 1 -- 172.21.15.5:0/3649158973 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54f0120d60 con 0x7f54f013cd80 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a090 0x7f54f013a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f6aca640 1 -- 172.21.15.5:0/3649158973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f54f009cc90 con 0x7f54f013a090 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f54e80a5640 con 0x7f54f013cd80 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a090 0x7f54f013a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 1 --2- 172.21.15.5:0/3649158973 >> v2:172.21.15.39:6812/1340838781 conn(0x7f54dc03b2a0 0x7f54dc03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient: get_auth_request con 0x7f54f013a090 auth_method 0 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f54e80e9e00 con 0x7f54f013cd80 2024-09-13T11:42:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54effff640 1 --2- 172.21.15.5:0/3649158973 >> v2:172.21.15.39:6812/1340838781 conn(0x7f54dc03b2a0 0x7f54dc03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54effff640 10 monclient: get_auth_request con 0x7f54dc03b2a0 auth_method 0 2024-09-13T11:42:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 10 monclient(hunting): handle_auth_done global_id 34105 payload 824 2024-09-13T11:42:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 1 -- 172.21.15.5:0/3649158973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54f00008e0 con 0x7f54f013a090 2024-09-13T11:42:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54f4866640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a090 0x7f54f013a470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f54e4010820 tx=0x7f54e4010ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e4018020 con 0x7f54f013a090 2024-09-13T11:42:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:42:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54ed7fa640 10 monclient: dump: 2024-09-13T11:42:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.727 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.699+0000 7f54effff640 1 --2- 172.21.15.5:0/3649158973 >> v2:172.21.15.39:6812/1340838781 conn(0x7f54dc03b2a0 0x7f54dc03d760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f54f013bb90 tx=0x7f54e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.703+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 <== mon.1 v2:172.21.15.39:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f54e4004240 con 0x7f54f013a090 2024-09-13T11:42:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.703+0000 7f54ed7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.703+0000 7f54ed7fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.703+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a090 msgr2=0x7f54f013a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.703+0000 7f54ed7fa640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a090 0x7f54f013a470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f54e4010820 tx=0x7f54e4010ce0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.703+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a090 msgr2=0x7f54f013a470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:42:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.703+0000 7f54ed7fa640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a090 0x7f54f013a470 secure :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f54e4010820 tx=0x7f54e4010ce0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f6aca640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T11:42:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f6aca640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a9b0 0x7f54f00a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f6aca640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f6aca640 1 -- 172.21.15.5:0/3649158973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f54f0136ee0 con 0x7f54f013a9b0 2024-09-13T11:42:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f4866640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a9b0 0x7f54f00a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f4866640 10 monclient: get_auth_request con 0x7f54f013a9b0 auth_method 0 2024-09-13T11:42:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f4866640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f4866640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f4866640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f4866640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f4866640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f4866640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f4866640 10 monclient(hunting): handle_auth_done global_id 34108 payload 824 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f4866640 1 -- 172.21.15.5:0/3649158973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f54f013adc0 con 0x7f54f013a9b0 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54f4866640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a9b0 0x7f54f00a1530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54e4010820 tx=0x7f54e40046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 <== mon.1 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e4018030 con 0x7f54f013a9b0 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.707+0000 7f54ed7fa640 10 monclient: dump: 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.730 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.731 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.731 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 <== mon.1 v2:172.21.15.39:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4530 (secure 0 0 0) 0x7f54e400eb70 con 0x7f54f013a9b0 2024-09-13T11:42:07.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54ed7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T11:42:07.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54ed7fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:42:07.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a9b0 msgr2=0x7f54f00a1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54ed7fa640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a9b0 0x7f54f00a1530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54e4010820 tx=0x7f54e40046d0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54ed7fa640 1 -- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a9b0 msgr2=0x7f54f00a1530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:42:07.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54ed7fa640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54f013a9b0 0x7f54f00a1530 secure :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54e4010820 tx=0x7f54e40046d0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.731 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 18, 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: "uptime": 775, 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:42:07.732 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2450521708007866372", 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.733 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540719139924082687", 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.734 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "59046471-79a9-4d90-9616-da0b446053a7", 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-13T11:39:20.750174Z", 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 19, 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "squid", 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 1, 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-13T11:42:07.735 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "reef", 2024-09-13T11:42:07.736 INFO:teuthology.orchestra.run.smithi005.stdout: "squid" 2024-09-13T11:42:07.737 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.737 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-13T11:42:07.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.737 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-13T11:42:07.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.737 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-13T11:42:07.737 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-13T11:42:07.737 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:42:07.737 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:42:07.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-13T11:42:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:3300", 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.739 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789", 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.39:6789/0", 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.39:6789/0", 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.741 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:42:07.742 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "squid", 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-13T11:42:07.743 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:42:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54f6aca640 1 -- 172.21.15.5:0/3649158973 >> v2:172.21.15.39:6812/1340838781 conn(0x7f54dc03b2a0 msgr2=0x7f54dc03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54f6aca640 1 --2- 172.21.15.5:0/3649158973 >> v2:172.21.15.39:6812/1340838781 conn(0x7f54dc03b2a0 0x7f54dc03d760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f54f013bb90 tx=0x7f54e0009040 comp rx=0 tx=0).stop 2024-09-13T11:42:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54f6aca640 10 monclient: shutdown 2024-09-13T11:42:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54f6aca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54f6aca640 1 -- 172.21.15.5:0/3649158973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54f013cd80 msgr2=0x7f54f0139b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54f6aca640 1 --2- 172.21.15.5:0/3649158973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54f013cd80 0x7f54f0139b50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f54e8019400 tx=0x7f54e8019590 comp rx=0 tx=0).stop 2024-09-13T11:42:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54f5067640 1 -- 172.21.15.5:0/3649158973 reap_dead start 2024-09-13T11:42:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54f5067640 1 -- 172.21.15.5:0/3649158973 reap_dead start 2024-09-13T11:42:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54f6aca640 1 -- 172.21.15.5:0/3649158973 shutdown_connections 2024-09-13T11:42:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54f6aca640 1 -- 172.21.15.5:0/3649158973 >> 172.21.15.5:0/3649158973 conn(0x7f54f00a6c70 msgr2=0x7f54f00a9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54f6aca640 1 -- 172.21.15.5:0/3649158973 shutdown_connections 2024-09-13T11:42:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.711+0000 7f54f6aca640 1 -- 172.21.15.5:0/3649158973 wait complete. 2024-09-13T11:42:07.746 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T11:42:07.746 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-13T11:42:07.746 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-13T11:42:07.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.851+0000 7fd5de166640 10 monclient: get_monmap_and_config 2024-09-13T11:42:07.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.851+0000 7fd5de166640 10 monclient: build_initial_monmap 2024-09-13T11:42:07.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.851+0000 7fd5de166640 10 monclient: monmap: 2024-09-13T11:42:07.856 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:07.856 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.856 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:07.855921+0000 2024-09-13T11:42:07.856 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:07.855921+0000 2024-09-13T11:42:07.856 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:07.857 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.851+0000 7fd5de166640 1 Processor -- start 2024-09-13T11:42:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.851+0000 7fd5de166640 1 -- start start 2024-09-13T11:42:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.851+0000 7fd5de166640 10 monclient: init 2024-09-13T11:42:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-13T11:42:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd5d80a0ea0 0x7fd5d80a1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 10 monclient(hunting): picked mon.noname-c con 0x7fd5d80a0ea0 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5d80a1ae0 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 0x7fd5d813d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 10 monclient(hunting): picked mon.noname-b con 0x7fd5d8135bc0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5d806f010 con 0x7fd5d80a1ae0 2024-09-13T11:42:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fd5d806f640 con 0x7fd5d80a0ea0 2024-09-13T11:42:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fd5d80a0000 con 0x7fd5d8135bc0 2024-09-13T11:42:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 10 monclient(hunting): authenticate will time out at 1131.204654s 2024-09-13T11:42:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 0x7fd5d813d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5ceffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd5d80a1ae0 0x7fd5d81354b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60676/0 (socket says 172.21.15.5:60676) 2024-09-13T11:42:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5ceffd640 1 -- 172.21.15.5:0/1583359888 learned_addr learned my addr 172.21.15.5:0/1583359888 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:42:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 10 monclient(hunting): get_auth_request con 0x7fd5d8135bc0 auth_method 0 2024-09-13T11:42:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d77fe640 1 --2- 172.21.15.5:0/1583359888 >> v2:172.21.15.39:3300/0 conn(0x7fd5d80a0ea0 0x7fd5d80a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d77fe640 10 monclient(hunting): get_auth_request con 0x7fd5d80a0ea0 auth_method 0 2024-09-13T11:42:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 10 monclient(hunting): handle_auth_done global_id 4427 payload 274 2024-09-13T11:42:07.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 1 -- 172.21.15.5:0/1583359888 >> v2:172.21.15.39:3300/0 conn(0x7fd5d80a0ea0 msgr2=0x7fd5d80a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:07.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 1 --2- 172.21.15.5:0/1583359888 >> v2:172.21.15.39:3300/0 conn(0x7fd5d80a0ea0 0x7fd5d80a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 1 -- 172.21.15.5:0/1583359888 >> v1:172.21.15.5:6789/0 conn(0x7fd5d80a1ae0 legacy=0x7fd5d81354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 10 monclient: _finish_hunting 0 2024-09-13T11:42:07.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 1 monclient: found mon.noname-b 2024-09-13T11:42:07.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:07.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 1 -- 172.21.15.5:0/1583359888 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3942917279 0 0) 0x7fd5d806f010 con 0x7fd5d80a1ae0 2024-09-13T11:42:07.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 1 -- 172.21.15.5:0/1583359888 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d809fd80 con 0x7fd5d8135bc0 2024-09-13T11:42:07.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:42:07.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d7fff640 1 --2- 172.21.15.5:0/1583359888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 0x7fd5d813d810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009d90 tx=0x7fd5d001c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fab87116db56ee2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 1 -- 172.21.15.5:0/1583359888 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00b2070 con 0x7fd5d8135bc0 2024-09-13T11:42:07.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:42:07.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 10 monclient: dump: 2024-09-13T11:42:07.871 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.871 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.871 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.871 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.871 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.871 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.871 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.871 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.871 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 10 monclient: _finish_auth 0 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 1 -- 172.21.15.5:0/1583359888 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d00a57f0 con 0x7fd5d8135bc0 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 1 -- 172.21.15.5:0/1583359888 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00a5bc0 con 0x7fd5d8135bc0 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 5 monclient: authenticate success, global_id 4427 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 4 set_mon_vals no callback set 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5d6ffd640 10 monclient: dump: 2024-09-13T11:42:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.873 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 10 monclient: shutdown 2024-09-13T11:42:07.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:07.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 1 -- 172.21.15.5:0/1583359888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 msgr2=0x7fd5d813d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 1 --2- 172.21.15.5:0/1583359888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 0x7fd5d813d810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009d90 tx=0x7fd5d001c800 comp rx=0 tx=0).stop 2024-09-13T11:42:07.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 1 -- 172.21.15.5:0/1583359888 shutdown_connections 2024-09-13T11:42:07.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 1 --2- 172.21.15.5:0/1583359888 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 0x7fd5d813d810 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 1 --2- 172.21.15.5:0/1583359888 >> v2:172.21.15.39:3300/0 conn(0x7fd5d80a0ea0 0x7fd5d80a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.855+0000 7fd5de166640 1 -- 172.21.15.5:0/1583359888 >> 172.21.15.5:0/1583359888 conn(0x7fd5d80a2930 msgr2=0x7fd5d80a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 -- 172.21.15.5:0/1583359888 shutdown_connections 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 -- 172.21.15.5:0/1583359888 wait complete. 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient: build_initial_monmap 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient: monmap: 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:07.876 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 Processor -- start 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 -- start start 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient: init 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d80a0ea0 0x7fd5d813b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient(hunting): picked mon.noname-b con 0x7fd5d80a0ea0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5d80a1ae0 0x7fd5d81cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient(hunting): picked mon.noname-a con 0x7fd5d80a1ae0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 0x7fd5d8139240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient(hunting): picked mon.noname-c con 0x7fd5d8135bc0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:07.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d8140570 con 0x7fd5d80a1ae0 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d809f780 con 0x7fd5d80a0ea0 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fd5d81406f0 con 0x7fd5d8135bc0 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient(hunting): authenticate will time out at 1131.209357s 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5ceffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5d80a1ae0 0x7fd5d81cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 0x7fd5d8139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 0x7fd5d8139240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:54874/0 (socket says 172.21.15.5:54874) 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 1 -- 172.21.15.5:0/1073876130 learned_addr learned my addr 172.21.15.5:0/1073876130 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 10 monclient(hunting): get_auth_request con 0x7fd5d8135bc0 auth_method 0 2024-09-13T11:42:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5ceffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5d80a1ae0 0x7fd5d81cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42862/0 (socket says 172.21.15.5:42862) 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5ceffd640 10 monclient(hunting): get_auth_request con 0x7fd5d80a1ae0 auth_method 0 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d77fe640 1 --2- 172.21.15.5:0/1073876130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d80a0ea0 0x7fd5d813b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 10 monclient(hunting): handle_auth_done global_id 4430 payload 824 2024-09-13T11:42:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 1 -- 172.21.15.5:0/1073876130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d80a0ea0 msgr2=0x7fd5d813b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 1 --2- 172.21.15.5:0/1073876130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d80a0ea0 0x7fd5d813b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 1 -- 172.21.15.5:0/1073876130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5d80a1ae0 msgr2=0x7fd5d81cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 1 --2- 172.21.15.5:0/1073876130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5d80a1ae0 0x7fd5d81cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 10 monclient: _finish_hunting 0 2024-09-13T11:42:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 1 monclient: found mon.noname-c 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 1 -- 172.21.15.5:0/1073876130 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d813b4a0 con 0x7fd5d8135bc0 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d77fe640 10 monclient: get_auth_request con 0x7fd5d80a0ea0 auth_method 0 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d77fe640 1 --2- 172.21.15.5:0/1073876130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d80a0ea0 0x7fd5d813b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d7fff640 1 --2- 172.21.15.5:0/1073876130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 0x7fd5d8139240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd5d00b0040 tx=0x7fd5d00195a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 1 -- 172.21.15.5:0/1073876130 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00b2050 con 0x7fd5d8135bc0 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 10 monclient: dump: 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.881 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 10 monclient: _finish_auth 0 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 5 monclient: authenticate success, global_id 4430 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 1 -- 172.21.15.5:0/1073876130 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d00be070 con 0x7fd5d8135bc0 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 1 -- 172.21.15.5:0/1073876130 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00ada70 con 0x7fd5d8135bc0 2024-09-13T11:42:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 10 monclient: dump: 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d67fc640 4 set_mon_vals no callback set 2024-09-13T11:42:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient: _renew_subs 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 -- 172.21.15.5:0/1073876130 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d813c240 con 0x7fd5d8135bc0 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient: _renew_subs 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 -- 172.21.15.5:0/1073876130 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d813cdd0 con 0x7fd5d8135bc0 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5de166640 1 -- 172.21.15.5:0/1073876130 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5d806dbf0 con 0x7fd5d8135bc0 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 1 -- 172.21.15.5:0/1073876130 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fd5d00a5670 con 0x7fd5d8135bc0 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 1 --2- 172.21.15.5:0/1073876130 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd5b803b2a0 0x7fd5b803d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.859+0000 7fd5d4ff9640 1 -- 172.21.15.5:0/1073876130 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7fd5d00e9ce0 con 0x7fd5d8135bc0 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.863+0000 7fd5d77fe640 1 --2- 172.21.15.5:0/1073876130 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd5b803b2a0 0x7fd5b803d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:07.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.863+0000 7fd5d77fe640 10 monclient: get_auth_request con 0x7fd5b803b2a0 auth_method 0 2024-09-13T11:42:07.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.863+0000 7fd5d77fe640 1 --2- 172.21.15.5:0/1073876130 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd5b803b2a0 0x7fd5b803d760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8009840 tx=0x7fd5c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:07.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.867+0000 7fd5d4ff9640 1 -- 172.21.15.5:0/1073876130 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fd5d0016540 con 0x7fd5d8135bc0 2024-09-13T11:42:07.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.867+0000 7fd5d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:07.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:07.867+0000 7fd5d4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.095+0000 7fd5de166640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-13T11:42:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.095+0000 7fd5de166640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-13T11:42:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.095+0000 7fd5de166640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.095+0000 7fd5de166640 1 -- 172.21.15.5:0/1073876130 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd5d80a1280 con 0x7fd5d8135bc0 2024-09-13T11:42:08.103 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.099+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:42:08.103 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.099+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=10,logm=66} crc {auth=3586118482,config=1618278320,health=3897446751,kv=3518782561,logm=3192206719}) 2024-09-13T11:42:08.103 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.099+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=11,kv=10,logm=67} crc {auth=3586118482,config=1618278320,health=588522927,kv=3518782561,logm=223382251}) 2024-09-13T11:42:08.103 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.099+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:42:08.103 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.099+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=10,logm=66} crc {auth=3586118482,config=1618278320,health=3897446751,kv=3518782561,logm=3192206719}) 2024-09-13T11:42:08.103 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.099+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=11,kv=9,logm=69} crc {auth=1473520871,config=1618278320,health=1522691141,kv=2968370512,logm=191099396}) 2024-09-13T11:42:08.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.099+0000 7fd5d4ff9640 1 -- 172.21.15.5:0/1073876130 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd5d00c3980 con 0x7fd5d8135bc0 2024-09-13T11:42:08.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.099+0000 7fd5d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-13T11:42:08.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.099+0000 7fd5d4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:42:08.104 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.099+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:42:08.104 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.099+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456375929}) 2024-09-13T11:42:08.104 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.099+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2559134604}) 2024-09-13T11:42:08.104 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.099+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:42:08.104 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.099+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456375929}) 2024-09-13T11:42:08.104 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.099+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1230496804}) 2024-09-13T11:42:08.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 1 -- 172.21.15.5:0/1073876130 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd5b803b2a0 msgr2=0x7fd5b803d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 1 --2- 172.21.15.5:0/1073876130 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd5b803b2a0 0x7fd5b803d760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8009840 tx=0x7fd5c8008040 comp rx=0 tx=0).stop 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 10 monclient: shutdown 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 1 -- 172.21.15.5:0/1073876130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 msgr2=0x7fd5d8139240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 1 --2- 172.21.15.5:0/1073876130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 0x7fd5d8139240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd5d00b0040 tx=0x7fd5d00195a0 comp rx=0 tx=0).stop 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 1 -- 172.21.15.5:0/1073876130 shutdown_connections 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 1 --2- 172.21.15.5:0/1073876130 >> v2:172.21.15.39:6812/1340838781 conn(0x7fd5b803b2a0 0x7fd5b803d760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 1 --2- 172.21.15.5:0/1073876130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5d8135bc0 0x7fd5d8139240 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 1 --2- 172.21.15.5:0/1073876130 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5d80a1ae0 0x7fd5d81cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 1 --2- 172.21.15.5:0/1073876130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d80a0ea0 0x7fd5d813b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 1 -- 172.21.15.5:0/1073876130 >> 172.21.15.5:0/1073876130 conn(0x7fd5d80a2930 msgr2=0x7fd5d8072d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 1 -- 172.21.15.5:0/1073876130 shutdown_connections 2024-09-13T11:42:08.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:08.103+0000 7fd5de166640 1 -- 172.21.15.5:0/1073876130 wait complete. 2024-09-13T11:42:08.107 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.103+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:42:08.107 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.103+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961239492}) 2024-09-13T11:42:08.107 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.103+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=887142931}) 2024-09-13T11:42:08.108 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.103+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:42:08.108 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.103+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961239492}) 2024-09-13T11:42:08.108 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.103+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=235598262}) 2024-09-13T11:42:08.108 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.103+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:42:08.108 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.103+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3046118397}) 2024-09-13T11:42:08.108 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.103+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842511759}) 2024-09-13T11:42:08.108 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.103+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:42:08.108 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.103+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3046118397}) 2024-09-13T11:42:08.108 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.103+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352750089}) 2024-09-13T11:42:08.109 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.107+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:42:08.109 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.107+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=57,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=7,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=938408173,mdsmap=2472758712,mgr=469291225,mgr_command_descs=2308730518,mgr_metadata=1309393946,mgrstat=3729862609,mon_config_key=3508062774,monmap=3781401897,nvmeofgw=598337686,osd_metadata=2350243558,osd_pg_creating=3722909383,osd_snap=723727393,osdmap=548152110}) 2024-09-13T11:42:08.110 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.107+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=58,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=8,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1} crc {logm=2309733508,mdsmap=2472758712,mgr=3589407732,mgr_command_descs=2308730518,mgr_metadata=1309393946,mgrstat=3729862609,mon_config_key=52290840,monmap=3781401897,nvmeofgw=598337686,osd_metadata=2459455437,osd_pg_creating=3722909383}) 2024-09-13T11:42:08.110 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.107+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:42:08.110 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.107+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=57,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=7,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=938408173,mdsmap=2472758712,mgr=469291225,mgr_command_descs=2308730518,mgr_metadata=1309393946,mgrstat=3729862609,mon_config_key=3508062774,monmap=3781401897,nvmeofgw=598337686,osd_metadata=2350243558,osd_pg_creating=3722909383,osd_snap=723727393,osdmap=548152110}) 2024-09-13T11:42:08.110 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.107+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=54,mdsmap=2,mgr=7,mgr_command_descs=1,mgrstat=7,mon_config_key=8,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=6} crc {logm=1145026167,mdsmap=3140514009,mgr=2845618932,mgr_command_descs=2308730518,mgrstat=437268701,mon_config_key=52290840,monmap=3781401897,nvmeofgw=598337686,osd_metadata=2459455437,osd_pg_creating=3722909383,osd_snap=723727393,osdmap=3323311655}) 2024-09-13T11:42:08.111 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.107+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:42:08.111 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.107+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=1957258817}) 2024-09-13T11:42:08.111 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.107+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=50} crc {osdmap=1167337624}) 2024-09-13T11:42:08.111 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.107+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T11:42:08.111 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.107+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=1957258817}) 2024-09-13T11:42:08.111 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-13T11:42:08.107+0000 7f096d796640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=51} crc {osdmap=473245511}) 2024-09-13T11:42:08.122 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-13T11:42:09.123 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T11:42:09.123 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T11:42:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 10 monclient: get_monmap_and_config 2024-09-13T11:42:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 10 monclient: build_initial_monmap 2024-09-13T11:42:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 10 monclient: monmap: 2024-09-13T11:42:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:09.258076+0000 2024-09-13T11:42:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:09.258076+0000 2024-09-13T11:42:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:09.259 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 1 Processor -- start 2024-09-13T11:42:09.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 1 -- start start 2024-09-13T11:42:09.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 10 monclient: init 2024-09-13T11:42:09.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:09.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-13T11:42:09.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f28ec137930 0x7f28ec09cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 10 monclient(hunting): picked mon.noname-c con 0x7f28ec137930 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28ec137d10 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28ec13cd80 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28ec0a5a80 con 0x7f28ec13cd80 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28ec09bef0 con 0x7f28ec137d10 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f28ec0a5fe0 con 0x7f28ec137930 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28f1947640 10 monclient(hunting): authenticate will time out at 1132.606870s 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28eb7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f28ec13cd80 0x7f28ec13f170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60678/0 (socket says 172.21.15.5:60678) 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28eb7fe640 1 -- 172.21.15.5:0/3559292360 learned_addr learned my addr 172.21.15.5:0/3559292360 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28eaffd640 1 --2- 172.21.15.5:0/3559292360 >> v2:172.21.15.39:3300/0 conn(0x7f28ec137930 0x7f28ec09cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28eaffd640 10 monclient(hunting): get_auth_request con 0x7f28ec137930 auth_method 0 2024-09-13T11:42:09.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 616618364 0 0) 0x7f28ec0a5a80 con 0x7f28ec13cd80 2024-09-13T11:42:09.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient(hunting): _negotiate 2024-09-13T11:42:09.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient(hunting): my global_id is 4587 2024-09-13T11:42:09.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28d4003530 con 0x7f28ec13cd80 2024-09-13T11:42:09.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:09.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2693553065 0 0) 0x7f28d4003530 con 0x7f28ec13cd80 2024-09-13T11:42:09.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 >> v2:172.21.15.39:3300/0 conn(0x7f28ec137930 msgr2=0x7f28ec09cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:09.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 --2- 172.21.15.5:0/3559292360 >> v2:172.21.15.39:3300/0 conn(0x7f28ec137930 0x7f28ec09cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 >> v1:172.21.15.39:6789/0 conn(0x7f28ec137d10 legacy=0x7f28ec1380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient: _finish_hunting 0 2024-09-13T11:42:09.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:42:09.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:09.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-13T11:42:09.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ec09c1d0 con 0x7f28ec13cd80 2024-09-13T11:42:09.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient: _finish_auth 0 2024-09-13T11:42:09.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient: _check_auth_tickets 2024-09-13T11:42:09.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:09.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3199362888 0 0) 0x7f28e4001a80 con 0x7f28ec13cd80 2024-09-13T11:42:09.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:42:09.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient: dump: 2024-09-13T11:42:09.266 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.266 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.266 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.267 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.267 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.267 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.267 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.267 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.267 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.267 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-13T11:42:09.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:09.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 >> v1:172.21.15.5:6789/0 conn(0x7f28ec13cd80 legacy=0x7f28ec13f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 --2- 172.21.15.5:0/3559292360 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 0x7f28d4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient(hunting): picked mon.a con 0x7f28d4003a60 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 --2- 172.21.15.5:0/3559292360 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28d4004810 0x7f28d4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient(hunting): picked mon.c con 0x7f28d4004810 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 --2- 172.21.15.5:0/3559292360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28d40053d0 0x7f28d400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient(hunting): picked mon.b con 0x7f28d40053d0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f28d400c210 con 0x7f28d4003a60 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f28d400c3c0 con 0x7f28d40053d0 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f28d400c5c0 con 0x7f28d4004810 2024-09-13T11:42:09.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28e9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.255+0000 7f28eaffd640 1 --2- 172.21.15.5:0/3559292360 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 0x7f28d4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 10 monclient(hunting): get_auth_request con 0x7f28d4003a60 auth_method 0 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28ea7fc640 1 --2- 172.21.15.5:0/3559292360 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28d4004810 0x7f28d4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28ea7fc640 10 monclient(hunting): get_auth_request con 0x7f28d4004810 auth_method 0 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28ea7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eb7fe640 1 --2- 172.21.15.5:0/3559292360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28d40053d0 0x7f28d400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 10 monclient(hunting): handle_auth_done global_id 4587 payload 293 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 1 -- 172.21.15.5:0/3559292360 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28d4004810 msgr2=0x7f28d4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 1 --2- 172.21.15.5:0/3559292360 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28d4004810 0x7f28d4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 1 -- 172.21.15.5:0/3559292360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28d40053d0 msgr2=0x7f28d400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 1 --2- 172.21.15.5:0/3559292360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28d40053d0 0x7f28d400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 10 monclient: _finish_hunting 0 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 1 monclient: found mon.a 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:09.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:09.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 1 -- 172.21.15.5:0/3559292360 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d4003530 con 0x7f28d4003a60 2024-09-13T11:42:09.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 10 monclient: _finish_auth 0 2024-09-13T11:42:09.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 10 monclient: _check_auth_tickets 2024-09-13T11:42:09.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:09.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eb7fe640 10 monclient: get_auth_request con 0x7f28d40053d0 auth_method 0 2024-09-13T11:42:09.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 5 monclient: authenticate success, global_id 4587 2024-09-13T11:42:09.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eb7fe640 1 --2- 172.21.15.5:0/3559292360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28d40053d0 0x7f28d400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:42:09.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:09.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eb7fe640 1 -- 172.21.15.5:0/3559292360 reap_dead start 2024-09-13T11:42:09.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28eaffd640 1 --2- 172.21.15.5:0/3559292360 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 0x7f28d4003e80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f28dc009ae0 tx=0x7f28dc01cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd257c64dba6eab server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:09.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc0096b0 con 0x7f28d4003a60 2024-09-13T11:42:09.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:09.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28e9ffb640 10 monclient: dump: 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28dc0a64d0 con 0x7f28d4003a60 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28e9ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:09.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28e9ffb640 1 -- 172.21.15.5:0/3559292360 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc0a68c0 con 0x7f28d4003a60 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 4 set_mon_vals no callback set 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28e9ffb640 10 monclient: dump: 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.279 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 10 monclient: shutdown 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 1 -- 172.21.15.5:0/3559292360 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 msgr2=0x7f28d4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 1 --2- 172.21.15.5:0/3559292360 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 0x7f28d4003e80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f28dc009ae0 tx=0x7f28dc01cae0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 1 -- 172.21.15.5:0/3559292360 shutdown_connections 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 1 --2- 172.21.15.5:0/3559292360 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 0x7f28d4003e80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 1 -- 172.21.15.5:0/3559292360 >> 172.21.15.5:0/3559292360 conn(0x7f28ec0a6c70 msgr2=0x7f28ec0a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 1 -- 172.21.15.5:0/3559292360 shutdown_connections 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 1 -- 172.21.15.5:0/3559292360 wait complete. 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 10 monclient: build_initial_monmap 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.259+0000 7f28f1947640 10 monclient: monmap: 2024-09-13T11:42:09.280 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 1 Processor -- start 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 1 -- start start 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient: init 2024-09-13T11:42:09.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 0x7f28ec1d0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient(hunting): picked mon.noname-a con 0x7f28d4003a60 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28ec136a90 0x7f28ec136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient(hunting): picked mon.noname-c con 0x7f28ec136a90 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28ec123680 0x7f28ec123af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient(hunting): picked mon.noname-b con 0x7f28ec123680 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec0a5280 con 0x7f28d4003a60 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec0a5400 con 0x7f28ec123680 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f28ec0a5580 con 0x7f28ec136a90 2024-09-13T11:42:09.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient(hunting): authenticate will time out at 1132.612795s 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 0x7f28ec1d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28ec136a90 0x7f28ec136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 0x7f28ec1d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42868/0 (socket says 172.21.15.5:42868) 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 1 -- 172.21.15.5:0/375558269 learned_addr learned my addr 172.21.15.5:0/375558269 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 10 monclient(hunting): get_auth_request con 0x7f28d4003a60 auth_method 0 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28ec136a90 0x7f28ec136ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:54880/0 (socket says 172.21.15.5:54880) 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 10 monclient(hunting): get_auth_request con 0x7f28ec136a90 auth_method 0 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eb7fe640 1 --2- 172.21.15.5:0/375558269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28ec123680 0x7f28ec123af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eb7fe640 10 monclient(hunting): get_auth_request con 0x7f28ec123680 auth_method 0 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 10 monclient(hunting): handle_auth_done global_id 4433 payload 824 2024-09-13T11:42:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 1 -- 172.21.15.5:0/375558269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28ec123680 msgr2=0x7f28ec123af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 1 --2- 172.21.15.5:0/375558269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28ec123680 0x7f28ec123af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 1 -- 172.21.15.5:0/375558269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 msgr2=0x7f28ec1d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 1 --2- 172.21.15.5:0/375558269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 0x7f28ec1d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 10 monclient: _finish_hunting 0 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 1 monclient: found mon.noname-c 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 1 -- 172.21.15.5:0/375558269 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ec0a9eb0 con 0x7f28ec136a90 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 1 --2- 172.21.15.5:0/375558269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 0x7f28ec1d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28ea7fc640 1 --2- 172.21.15.5:0/375558269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28ec136a90 0x7f28ec136ef0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28e000f850 tx=0x7f28e000fd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 1 -- 172.21.15.5:0/375558269 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e001a070 con 0x7f28ec136a90 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:42:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 10 monclient: dump: 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 10 monclient: _finish_auth 0 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 10 monclient: _check_auth_tickets 2024-09-13T11:42:09.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 1 -- 172.21.15.5:0/375558269 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e000fe40 con 0x7f28ec136a90 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 5 monclient: authenticate success, global_id 4433 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient: _renew_subs 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 1 -- 172.21.15.5:0/375558269 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28ec126c00 con 0x7f28ec136a90 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 1 -- 172.21.15.5:0/375558269 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e00067e0 con 0x7f28ec136a90 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 10 monclient: dump: 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.287 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28e97fa640 4 set_mon_vals no callback set 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient: _renew_subs 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 1 -- 172.21.15.5:0/375558269 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f28ec1d07e0 con 0x7f28ec136a90 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:09.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28f1947640 1 -- 172.21.15.5:0/375558269 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28ec0a5960 con 0x7f28ec136a90 2024-09-13T11:42:09.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 1 -- 172.21.15.5:0/375558269 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f28e001d070 con 0x7f28ec136a90 2024-09-13T11:42:09.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 1 --2- 172.21.15.5:0/375558269 >> v2:172.21.15.39:6812/1340838781 conn(0x7f28d803b2a0 0x7f28d803d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28d3fff640 1 -- 172.21.15.5:0/375558269 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f28e004e3e0 con 0x7f28ec136a90 2024-09-13T11:42:09.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 1 --2- 172.21.15.5:0/375558269 >> v2:172.21.15.39:6812/1340838781 conn(0x7f28d803b2a0 0x7f28d803d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.263+0000 7f28eaffd640 10 monclient: get_auth_request con 0x7f28d803b2a0 auth_method 0 2024-09-13T11:42:09.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.267+0000 7f28eaffd640 1 --2- 172.21.15.5:0/375558269 >> v2:172.21.15.39:6812/1340838781 conn(0x7f28d803b2a0 0x7f28d803d760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f28dc000c10 tx=0x7f28dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:09.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.267+0000 7f28d3fff640 1 -- 172.21.15.5:0/375558269 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f28ec126c00 con 0x7f28ec136a90 2024-09-13T11:42:09.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.267+0000 7f28d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:09.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.267+0000 7f28d3fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:09.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.499+0000 7f28f1947640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T11:42:09.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.499+0000 7f28f1947640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T11:42:09.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.499+0000 7f28f1947640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:09.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.499+0000 7f28f1947640 1 -- 172.21.15.5:0/375558269 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f28ec125ca0 con 0x7f28ec136a90 2024-09-13T11:42:09.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.499+0000 7f28d3fff640 1 -- 172.21.15.5:0/375558269 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1548 (secure 0 0 0) 0x7f28e001e3b0 con 0x7f28ec136a90 2024-09-13T11:42:09.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.499+0000 7f28d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T11:42:09.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.499+0000 7f28d3fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:42:09.505 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","modified":"2024-09-13T11:39:20.750174Z","created":"2024-09-13T11:39:20.750174Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T11:42:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 1 -- 172.21.15.5:0/375558269 >> v2:172.21.15.39:6812/1340838781 conn(0x7f28d803b2a0 msgr2=0x7f28d803d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 1 --2- 172.21.15.5:0/375558269 >> v2:172.21.15.39:6812/1340838781 conn(0x7f28d803b2a0 0x7f28d803d760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f28dc000c10 tx=0x7f28dc008040 comp rx=0 tx=0).stop 2024-09-13T11:42:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 10 monclient: shutdown 2024-09-13T11:42:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 1 -- 172.21.15.5:0/375558269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28ec136a90 msgr2=0x7f28ec136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 1 --2- 172.21.15.5:0/375558269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28ec136a90 0x7f28ec136ef0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28e000f850 tx=0x7f28e000fd10 comp rx=0 tx=0).stop 2024-09-13T11:42:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 1 -- 172.21.15.5:0/375558269 shutdown_connections 2024-09-13T11:42:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 1 --2- 172.21.15.5:0/375558269 >> v2:172.21.15.39:6812/1340838781 conn(0x7f28d803b2a0 0x7f28d803d760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 1 --2- 172.21.15.5:0/375558269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28ec123680 0x7f28ec123af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 1 --2- 172.21.15.5:0/375558269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28ec136a90 0x7f28ec136ef0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 1 --2- 172.21.15.5:0/375558269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28d4003a60 0x7f28ec1d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 1 -- 172.21.15.5:0/375558269 >> 172.21.15.5:0/375558269 conn(0x7f28ec0a6c70 msgr2=0x7f28ec0a9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:09.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 1 -- 172.21.15.5:0/375558269 shutdown_connections 2024-09-13T11:42:09.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.503+0000 7f28f1947640 1 -- 172.21.15.5:0/375558269 wait complete. 2024-09-13T11:42:09.522 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T11:42:09.522 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T11:42:09.531 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T11:42:09.532 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:42:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.635+0000 7f06ddcc8640 10 monclient: get_monmap_and_config 2024-09-13T11:42:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.635+0000 7f06ddcc8640 10 monclient: build_initial_monmap 2024-09-13T11:42:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.635+0000 7f06ddcc8640 10 monclient: monmap: 2024-09-13T11:42:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:09.640353+0000 2024-09-13T11:42:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:09.640353+0000 2024-09-13T11:42:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:09.641 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 1 Processor -- start 2024-09-13T11:42:09.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 1 -- start start 2024-09-13T11:42:09.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 10 monclient: init 2024-09-13T11:42:09.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:09.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-13T11:42:09.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f06d8073620 0x7f06d8137a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 10 monclient(hunting): picked mon.noname-a con 0x7f06d8073620 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:09.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06d8073a00 0x7f06d809fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 10 monclient(hunting): picked mon.noname-b con 0x7f06d8073a00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:09.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06d8073de0 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06d80a1da0 con 0x7f06d8073de0 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f06d80a2690 con 0x7f06d8073620 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f06d806fc20 con 0x7f06d8073a00 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06ddcc8640 10 monclient(hunting): authenticate will time out at 1132.989099s 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f06d8073620 0x7f06d8137a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d6ffd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06d8073a00 0x7f06d809fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f06d8073620 0x7f06d8137a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42872/0 (socket says 172.21.15.5:42872) 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 1 -- 172.21.15.5:0/2109704748 learned_addr learned my addr 172.21.15.5:0/2109704748 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 10 monclient(hunting): get_auth_request con 0x7f06d8073620 auth_method 0 2024-09-13T11:42:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d6ffd640 10 monclient(hunting): get_auth_request con 0x7f06d8073a00 auth_method 0 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:09.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 10 monclient(hunting): handle_auth_done global_id 4593 payload 274 2024-09-13T11:42:09.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 1 -- 172.21.15.5:0/2109704748 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06d8073a00 msgr2=0x7f06d809fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:09.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 1 --2- 172.21.15.5:0/2109704748 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06d8073a00 0x7f06d809fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 1 -- 172.21.15.5:0/2109704748 >> v1:172.21.15.39:6789/0 conn(0x7f06d8073de0 legacy=0x7f06d80a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 10 monclient: _finish_hunting 0 2024-09-13T11:42:09.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 1 monclient: found mon.noname-a 2024-09-13T11:42:09.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:09.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:09.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 1 -- 172.21.15.5:0/2109704748 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d80a1b20 con 0x7f06d8073620 2024-09-13T11:42:09.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d6ffd640 1 --2- 172.21.15.5:0/2109704748 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06d8073a00 0x7f06d809fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:09.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d77fe640 1 --2- 172.21.15.5:0/2109704748 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06d8073620 0x7f06d8137a70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009870 tx=0x7f06c801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b614d6dacbfaf444 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.639+0000 7f06d67fc640 1 -- 172.21.15.5:0/2109704748 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80b0070 con 0x7f06d8073620 2024-09-13T11:42:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d67fc640 10 monclient: dump: 2024-09-13T11:42:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d67fc640 10 monclient: _finish_auth 0 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d67fc640 10 monclient: _check_auth_tickets 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d67fc640 1 -- 172.21.15.5:0/2109704748 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c80ab600 con 0x7f06d8073620 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d67fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d67fc640 1 -- 172.21.15.5:0/2109704748 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80ab9d0 con 0x7f06d8073620 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 5 monclient: authenticate success, global_id 4593 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 4 set_mon_vals no callback set 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d67fc640 10 monclient: dump: 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: shutdown 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 -- 172.21.15.5:0/2109704748 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06d8073620 msgr2=0x7f06d8137a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 --2- 172.21.15.5:0/2109704748 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06d8073620 0x7f06d8137a70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009870 tx=0x7f06c801c3c0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 -- 172.21.15.5:0/2109704748 shutdown_connections 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 --2- 172.21.15.5:0/2109704748 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06d8073a00 0x7f06d809fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 --2- 172.21.15.5:0/2109704748 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06d8073620 0x7f06d8137a70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 -- 172.21.15.5:0/2109704748 >> 172.21.15.5:0/2109704748 conn(0x7f06d80a3710 msgr2=0x7f06d80a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 -- 172.21.15.5:0/2109704748 shutdown_connections 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 -- 172.21.15.5:0/2109704748 wait complete. 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: build_initial_monmap 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: monmap: 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 Processor -- start 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 -- start start 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: init 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06d8073620 0x7f06d81cf670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient(hunting): picked mon.noname-c con 0x7f06d8073620 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06d8073a00 0x7f06d8133230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient(hunting): picked mon.noname-a con 0x7f06d8073a00 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06d8073de0 0x7f06d813c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient(hunting): picked mon.noname-b con 0x7f06d8073de0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f06d8142150 con 0x7f06d8073a00 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f06d80a0cb0 con 0x7f06d8073de0 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f06d81422d0 con 0x7f06d8073620 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient(hunting): authenticate will time out at 1132.994047s 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06d8073a00 0x7f06d8133230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06d8073a00 0x7f06d8133230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42886/0 (socket says 172.21.15.5:42886) 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d77fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06d8073620 0x7f06d81cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 1 -- 172.21.15.5:0/42142790 learned_addr learned my addr 172.21.15.5:0/42142790 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 10 monclient(hunting): get_auth_request con 0x7f06d8073a00 auth_method 0 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d77fe640 10 monclient(hunting): get_auth_request con 0x7f06d8073620 auth_method 0 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d7fff640 1 --2- 172.21.15.5:0/42142790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06d8073de0 0x7f06d813c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d7fff640 10 monclient(hunting): get_auth_request con 0x7f06d8073de0 auth_method 0 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 10 monclient(hunting): handle_auth_done global_id 4596 payload 824 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 1 -- 172.21.15.5:0/42142790 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06d8073620 msgr2=0x7f06d81cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 1 --2- 172.21.15.5:0/42142790 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06d8073620 0x7f06d81cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 1 -- 172.21.15.5:0/42142790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06d8073de0 msgr2=0x7f06d813c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 1 --2- 172.21.15.5:0/42142790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06d8073de0 0x7f06d813c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 10 monclient: _finish_hunting 0 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 1 monclient: found mon.noname-a 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 1 -- 172.21.15.5:0/42142790 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d80a4a50 con 0x7f06d8073a00 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d77fe640 1 --2- 172.21.15.5:0/42142790 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06d8073620 0x7f06d81cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d6ffd640 1 --2- 172.21.15.5:0/42142790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06d8073a00 0x7f06d8133230 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f06cc00ebe0 tx=0x7f06cc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 1 -- 172.21.15.5:0/42142790 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06cc017070 con 0x7f06d8073a00 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 10 monclient: dump: 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 10 monclient: _finish_auth 0 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 10 monclient: _check_auth_tickets 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 1 -- 172.21.15.5:0/42142790 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06cc00cbd0 con 0x7f06d8073a00 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 5 monclient: authenticate success, global_id 4596 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: _renew_subs 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 -- 172.21.15.5:0/42142790 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d8074a10 con 0x7f06d8073a00 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 1 -- 172.21.15.5:0/42142790 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06cc013470 con 0x7f06d8073a00 2024-09-13T11:42:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 10 monclient: dump: 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: _renew_subs 2024-09-13T11:42:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06d5ffb640 4 set_mon_vals no callback set 2024-09-13T11:42:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 -- 172.21.15.5:0/42142790 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d813e2d0 con 0x7f06d8073a00 2024-09-13T11:42:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06ddcc8640 1 -- 172.21.15.5:0/42142790 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d8137a70 con 0x7f06d8073a00 2024-09-13T11:42:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.643+0000 7f06bffff640 1 -- 172.21.15.5:0/42142790 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f06cc01a070 con 0x7f06d8073a00 2024-09-13T11:42:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.647+0000 7f06bffff640 1 --2- 172.21.15.5:0/42142790 >> v2:172.21.15.39:6812/1340838781 conn(0x7f06c403b2d0 0x7f06c403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.647+0000 7f06bffff640 1 -- 172.21.15.5:0/42142790 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f06cc04ebf0 con 0x7f06d8073a00 2024-09-13T11:42:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.647+0000 7f06d77fe640 1 --2- 172.21.15.5:0/42142790 >> v2:172.21.15.39:6812/1340838781 conn(0x7f06c403b2d0 0x7f06c403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.647+0000 7f06d77fe640 10 monclient: get_auth_request con 0x7f06c403b2d0 auth_method 0 2024-09-13T11:42:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.647+0000 7f06d77fe640 1 --2- 172.21.15.5:0/42142790 >> v2:172.21.15.39:6812/1340838781 conn(0x7f06c403b2d0 0x7f06c403d790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f06c80ad000 tx=0x7f06c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:09.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.651+0000 7f06bffff640 1 -- 172.21.15.5:0/42142790 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f06d8074a10 con 0x7f06d8073a00 2024-09-13T11:42:09.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.651+0000 7f06bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:09.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.651+0000 7f06bffff640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:09.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.811+0000 7f06ddcc8640 1 -- 172.21.15.5:0/42142790 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f06d8069ff0 con 0x7f06c403b2d0 2024-09-13T11:42:09.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.811+0000 7f06bffff640 1 -- 172.21.15.5:0/42142790 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33921 (secure 0 0 0) 0x7f06d8069ff0 con 0x7f06c403b2d0 2024-09-13T11:42:09.817 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:42:09.818 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T11:42:09.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 1 -- 172.21.15.5:0/42142790 >> v2:172.21.15.39:6812/1340838781 conn(0x7f06c403b2d0 msgr2=0x7f06c403d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 1 --2- 172.21.15.5:0/42142790 >> v2:172.21.15.39:6812/1340838781 conn(0x7f06c403b2d0 0x7f06c403d790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f06c80ad000 tx=0x7f06c8008040 comp rx=0 tx=0).stop 2024-09-13T11:42:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 10 monclient: shutdown 2024-09-13T11:42:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 1 -- 172.21.15.5:0/42142790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06d8073a00 msgr2=0x7f06d8133230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 1 --2- 172.21.15.5:0/42142790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06d8073a00 0x7f06d8133230 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f06cc00ebe0 tx=0x7f06cc00c520 comp rx=0 tx=0).stop 2024-09-13T11:42:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 1 -- 172.21.15.5:0/42142790 shutdown_connections 2024-09-13T11:42:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 1 --2- 172.21.15.5:0/42142790 >> v2:172.21.15.39:6812/1340838781 conn(0x7f06c403b2d0 0x7f06c403d790 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 1 --2- 172.21.15.5:0/42142790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06d8073de0 0x7f06d813c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 1 --2- 172.21.15.5:0/42142790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06d8073a00 0x7f06d8133230 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 1 --2- 172.21.15.5:0/42142790 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f06d8073620 0x7f06d81cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 1 -- 172.21.15.5:0/42142790 >> 172.21.15.5:0/42142790 conn(0x7f06d80a3710 msgr2=0x7f06d8135d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 1 -- 172.21.15.5:0/42142790 shutdown_connections 2024-09-13T11:42:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.815+0000 7f06ddcc8640 1 -- 172.21.15.5:0/42142790 wait complete. 2024-09-13T11:42:09.833 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-13T11:42:08.329057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167812,"kb_used_data":1464,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063228,"statfs":{"total":579820584960,"available":579648745472,"internally_reserved":0,"allocated":1499136,"data_stored":1545562,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003007"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.252094+0000","last_change":"2024-09-13T11:40:14.135514+0000","last_active":"2024-09-13T11:40:41.252094+0000","last_peered":"2024-09-13T11:40:41.252094+0000","last_clean":"2024-09-13T11:40:41.252094+0000","last_became_active":"2024-09-13T11:40:12.461041+0000","last_became_peered":"2024-09-13T11:40:12.461041+0000","last_unstale":"2024-09-13T11:40:41.252094+0000","last_undegraded":"2024-09-13T11:40:41.252094+0000","last_fullsized":"2024-09-13T11:40:41.252094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:50.409094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126878,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.466426+0000","last_change":"2024-09-13T11:40:14.143285+0000","last_active":"2024-09-13T11:40:41.466426+0000","last_peered":"2024-09-13T11:40:41.466426+0000","last_clean":"2024-09-13T11:40:41.466426+0000","last_became_active":"2024-09-13T11:40:12.106972+0000","last_became_peered":"2024-09-13T11:40:12.106972+0000","last_unstale":"2024-09-13T11:40:41.466426+0000","last_undegraded":"2024-09-13T11:40:41.466426+0000","last_fullsized":"2024-09-13T11:40:41.466426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:29.565401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001297091,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.252642+0000","last_change":"2024-09-13T11:40:14.135429+0000","last_active":"2024-09-13T11:40:41.252642+0000","last_peered":"2024-09-13T11:40:41.252642+0000","last_clean":"2024-09-13T11:40:41.252642+0000","last_became_active":"2024-09-13T11:40:12.120816+0000","last_became_peered":"2024-09-13T11:40:12.120816+0000","last_unstale":"2024-09-13T11:40:41.252642+0000","last_undegraded":"2024-09-13T11:40:41.252642+0000","last_fullsized":"2024-09-13T11:40:41.252642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:42.012635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011926969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.466384+0000","last_change":"2024-09-13T11:40:14.143208+0000","last_active":"2024-09-13T11:40:41.466384+0000","last_peered":"2024-09-13T11:40:41.466384+0000","last_clean":"2024-09-13T11:40:41.466384+0000","last_became_active":"2024-09-13T11:40:12.120092+0000","last_became_peered":"2024-09-13T11:40:12.120092+0000","last_unstale":"2024-09-13T11:40:41.466384+0000","last_undegraded":"2024-09-13T11:40:41.466384+0000","last_fullsized":"2024-09-13T11:40:41.466384+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:10.230543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001217161,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":36,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.461936+0000","last_change":"2024-09-13T11:40:14.139720+0000","last_active":"2024-09-13T11:40:41.461936+0000","last_peered":"2024-09-13T11:40:41.461936+0000","last_clean":"2024-09-13T11:40:41.461936+0000","last_became_active":"2024-09-13T11:40:12.110703+0000","last_became_peered":"2024-09-13T11:40:12.110703+0000","last_unstale":"2024-09-13T11:40:41.461936+0000","last_undegraded":"2024-09-13T11:40:41.461936+0000","last_fullsized":"2024-09-13T11:40:41.461936+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:57.779487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021711669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.250163+0000","last_change":"2024-09-13T11:40:14.131038+0000","last_active":"2024-09-13T11:40:41.250163+0000","last_peered":"2024-09-13T11:40:41.250163+0000","last_clean":"2024-09-13T11:40:41.250163+0000","last_became_active":"2024-09-13T11:40:12.102142+0000","last_became_peered":"2024-09-13T11:40:12.102142+0000","last_unstale":"2024-09-13T11:40:41.250163+0000","last_undegraded":"2024-09-13T11:40:41.250163+0000","last_fullsized":"2024-09-13T11:40:41.250163+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:15.679795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046429,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.251587+0000","last_change":"2024-09-13T11:40:14.135976+0000","last_active":"2024-09-13T11:40:41.251587+0000","last_peered":"2024-09-13T11:40:41.251587+0000","last_clean":"2024-09-13T11:40:41.251587+0000","last_became_active":"2024-09-13T11:40:12.105280+0000","last_became_peered":"2024-09-13T11:40:12.105280+0000","last_unstale":"2024-09-13T11:40:41.251587+0000","last_undegraded":"2024-09-13T11:40:41.251587+0000","last_fullsized":"2024-09-13T11:40:41.251587+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:20.791570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001476088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.462629+0000","last_change":"2024-09-13T11:40:14.138397+0000","last_active":"2024-09-13T11:40:41.462629+0000","last_peered":"2024-09-13T11:40:41.462629+0000","last_clean":"2024-09-13T11:40:41.462629+0000","last_became_active":"2024-09-13T11:40:12.113752+0000","last_became_peered":"2024-09-13T11:40:12.113752+0000","last_unstale":"2024-09-13T11:40:41.462629+0000","last_undegraded":"2024-09-13T11:40:41.462629+0000","last_fullsized":"2024-09-13T11:40:41.462629+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:50:52.508777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086960399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":125,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.251870+0000","last_change":"2024-09-13T11:40:05.394873+0000","last_active":"2024-09-13T11:40:41.251870+0000","last_peered":"2024-09-13T11:40:41.251870+0000","last_clean":"2024-09-13T11:40:41.251870+0000","last_became_active":"2024-09-13T11:40:05.394176+0000","last_became_peered":"2024-09-13T11:40:05.394176+0000","last_unstale":"2024-09-13T11:40:41.251870+0000","last_undegraded":"2024-09-13T11:40:41.251870+0000","last_fullsized":"2024-09-13T11:40:41.251870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_clean_scrub_stamp":"2024-09-13T11:40:04.377013+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:00:31.498589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607579,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":81297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607579,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":229376,"data_stored":81278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607579,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":286720,"data_stored":610206,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607579,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":81297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640284,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":229376,"data_stored":81278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640284,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":610206,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:42:09.835 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:42:09.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient: get_monmap_and_config 2024-09-13T11:42:09.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient: build_initial_monmap 2024-09-13T11:42:09.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient: monmap: 2024-09-13T11:42:09.954 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:09.954 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.954 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:09.954092+0000 2024-09-13T11:42:09.954 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:09.954092+0000 2024-09-13T11:42:09.954 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:09.955 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.955 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:09.955 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:09.955 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:09.955 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:09.955 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:09.955 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 1 Processor -- start 2024-09-13T11:42:09.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 1 -- start start 2024-09-13T11:42:09.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient: init 2024-09-13T11:42:09.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:09.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-13T11:42:09.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa490134810 0x7fa490134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient(hunting): picked mon.noname-b con 0x7fa490134810 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa49009d1d0 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa49009dcc0 0x7fa49013c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient(hunting): picked mon.noname-a con 0x7fa49009dcc0 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa49009ce60 con 0x7fa49009d1d0 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa4900a0950 con 0x7fa49009dcc0 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa49009eeb0 con 0x7fa490134810 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa498713640 10 monclient(hunting): authenticate will time out at 1133.302972s 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa49009dcc0 0x7fa49013c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa4964af640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa490134810 0x7fa490134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa49009dcc0 0x7fa49013c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42898/0 (socket says 172.21.15.5:42898) 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa4964af640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa490134810 0x7fa490134bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:54912/0 (socket says 172.21.15.5:54912) 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 1 -- 172.21.15.5:0/3583033196 learned_addr learned my addr 172.21.15.5:0/3583033196 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 10 monclient(hunting): get_auth_request con 0x7fa49009dcc0 auth_method 0 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa4964af640 10 monclient(hunting): get_auth_request con 0x7fa490134810 auth_method 0 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa4964af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa4964af640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa4964af640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa4964af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa4964af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa4964af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 10 monclient(hunting): handle_auth_done global_id 4599 payload 274 2024-09-13T11:42:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 1 -- 172.21.15.5:0/3583033196 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa490134810 msgr2=0x7fa490134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 1 --2- 172.21.15.5:0/3583033196 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa490134810 0x7fa490134bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 1 -- 172.21.15.5:0/3583033196 >> v1:172.21.15.5:6789/0 conn(0x7fa49009d1d0 legacy=0x7fa49009d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 10 monclient: _finish_hunting 0 2024-09-13T11:42:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 1 monclient: found mon.noname-a 2024-09-13T11:42:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa496cb0640 1 -- 172.21.15.5:0/3583033196 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa490135130 con 0x7fa49009dcc0 2024-09-13T11:42:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.951+0000 7fa4964af640 1 --2- 172.21.15.5:0/3583033196 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa490134810 0x7fa490134bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa496cb0640 1 --2- 172.21.15.5:0/3583033196 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa49009dcc0 0x7fa49013c690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa488009870 tx=0x7fa48801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6d717cd4b5d8095 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 1 -- 172.21.15.5:0/3583033196 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4880b0070 con 0x7fa49009dcc0 2024-09-13T11:42:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 10 monclient: dump: 2024-09-13T11:42:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 10 monclient: _finish_auth 0 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 10 monclient: _check_auth_tickets 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 5 monclient: authenticate success, global_id 4599 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 1 -- 172.21.15.5:0/3583033196 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4880ab600 con 0x7fa49009dcc0 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 1 -- 172.21.15.5:0/3583033196 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4880ab9d0 con 0x7fa49009dcc0 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 4 set_mon_vals no callback set 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa4954ad640 10 monclient: dump: 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.967 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 10 monclient: shutdown 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 1 -- 172.21.15.5:0/3583033196 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa49009dcc0 msgr2=0x7fa49013c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 1 --2- 172.21.15.5:0/3583033196 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa49009dcc0 0x7fa49013c690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa488009870 tx=0x7fa48801c420 comp rx=0 tx=0).stop 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 1 -- 172.21.15.5:0/3583033196 shutdown_connections 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 1 --2- 172.21.15.5:0/3583033196 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa49009dcc0 0x7fa49013c690 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 1 --2- 172.21.15.5:0/3583033196 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa490134810 0x7fa490134bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 1 -- 172.21.15.5:0/3583033196 >> 172.21.15.5:0/3583033196 conn(0x7fa4900a2930 msgr2=0x7fa4900a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 1 -- 172.21.15.5:0/3583033196 shutdown_connections 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 1 -- 172.21.15.5:0/3583033196 wait complete. 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 10 monclient: build_initial_monmap 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 10 monclient: monmap: 2024-09-13T11:42:09.968 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 1 Processor -- start 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 1 -- start start 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.955+0000 7fa498713640 10 monclient: init 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:09.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa49009d1d0 0x7fa490132d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient(hunting): picked mon.noname-b con 0x7fa49009d1d0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa49009dcc0 0x7fa490133280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient(hunting): picked mon.noname-c con 0x7fa49009dcc0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa490134810 0x7fa4901337c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient(hunting): picked mon.noname-a con 0x7fa490134810 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa49009e360 con 0x7fa490134810 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa4900a0d30 con 0x7fa49009d1d0 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa49013f3f0 con 0x7fa49009dcc0 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient(hunting): authenticate will time out at 1133.308373s 2024-09-13T11:42:09.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa495cae640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa49009dcc0 0x7fa490133280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa490134810 0x7fa4901337c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa495cae640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa49009dcc0 0x7fa490133280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:54924/0 (socket says 172.21.15.5:54924) 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa495cae640 1 -- 172.21.15.5:0/3074800970 learned_addr learned my addr 172.21.15.5:0/3074800970 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa495cae640 10 monclient(hunting): get_auth_request con 0x7fa49009dcc0 auth_method 0 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa495cae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa495cae640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa495cae640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 10 monclient(hunting): get_auth_request con 0x7fa490134810 auth_method 0 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa495cae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa495cae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa495cae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa4964af640 1 --2- 172.21.15.5:0/3074800970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa49009d1d0 0x7fa490132d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 10 monclient(hunting): handle_auth_done global_id 4602 payload 824 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 1 -- 172.21.15.5:0/3074800970 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa49009dcc0 msgr2=0x7fa490133280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 1 --2- 172.21.15.5:0/3074800970 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa49009dcc0 0x7fa490133280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 1 -- 172.21.15.5:0/3074800970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa49009d1d0 msgr2=0x7fa490132d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 1 --2- 172.21.15.5:0/3074800970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa49009d1d0 0x7fa490132d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 10 monclient: _finish_hunting 0 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 1 monclient: found mon.noname-a 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 1 -- 172.21.15.5:0/3074800970 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49011e7b0 con 0x7fa490134810 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa4964af640 10 monclient: get_auth_request con 0x7fa49009d1d0 auth_method 0 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa4964af640 1 --2- 172.21.15.5:0/3074800970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa49009d1d0 0x7fa490132d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa496cb0640 1 --2- 172.21.15.5:0/3074800970 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa490134810 0x7fa4901337c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa4880ab000 tx=0x7fa4880191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 1 -- 172.21.15.5:0/3074800970 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4880b0040 con 0x7fa490134810 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 10 monclient: dump: 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.973 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.974 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.974 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.974 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.974 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 10 monclient: _finish_auth 0 2024-09-13T11:42:09.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 10 monclient: _check_auth_tickets 2024-09-13T11:42:09.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:09.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 5 monclient: authenticate success, global_id 4602 2024-09-13T11:42:09.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient: _renew_subs 2024-09-13T11:42:09.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 1 -- 172.21.15.5:0/3074800970 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4880b4420 con 0x7fa490134810 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 1 -- 172.21.15.5:0/3074800970 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa49011ea90 con 0x7fa490134810 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 1 -- 172.21.15.5:0/3074800970 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4880b4810 con 0x7fa490134810 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 10 monclient: dump: 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:09.975 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa494cac640 4 set_mon_vals no callback set 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient: _renew_subs 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 1 -- 172.21.15.5:0/3074800970 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa490073c20 con 0x7fa490134810 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa498713640 1 -- 172.21.15.5:0/3074800970 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa490134bf0 con 0x7fa490134810 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 1 -- 172.21.15.5:0/3074800970 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fa4880b49b0 con 0x7fa490134810 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 1 --2- 172.21.15.5:0/3074800970 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa47c03b2a0 0x7fa47c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa48effd640 1 -- 172.21.15.5:0/3074800970 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7fa4880e88a0 con 0x7fa490134810 2024-09-13T11:42:09.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa4964af640 1 --2- 172.21.15.5:0/3074800970 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa47c03b2a0 0x7fa47c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:09.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa4964af640 10 monclient: get_auth_request con 0x7fa47c03b2a0 auth_method 0 2024-09-13T11:42:09.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.959+0000 7fa4964af640 1 --2- 172.21.15.5:0/3074800970 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa47c03b2a0 0x7fa47c03d760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa4840074d0 tx=0x7fa48400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:09.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.963+0000 7fa48effd640 1 -- 172.21.15.5:0/3074800970 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fa488016130 con 0x7fa490134810 2024-09-13T11:42:09.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.963+0000 7fa48effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:09.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:09.963+0000 7fa48effd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:10.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.139+0000 7fa498713640 1 -- 172.21.15.5:0/3074800970 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa49011d570 con 0x7fa47c03b2a0 2024-09-13T11:42:10.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.139+0000 7fa48effd640 1 -- 172.21.15.5:0/3074800970 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33921 (secure 0 0 0) 0x7fa49011d570 con 0x7fa47c03b2a0 2024-09-13T11:42:10.143 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:42:10.144 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T11:42:10.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 1 -- 172.21.15.5:0/3074800970 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa47c03b2a0 msgr2=0x7fa47c03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 1 --2- 172.21.15.5:0/3074800970 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa47c03b2a0 0x7fa47c03d760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa4840074d0 tx=0x7fa48400b040 comp rx=0 tx=0).stop 2024-09-13T11:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 10 monclient: shutdown 2024-09-13T11:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 1 -- 172.21.15.5:0/3074800970 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa490134810 msgr2=0x7fa4901337c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 1 --2- 172.21.15.5:0/3074800970 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa490134810 0x7fa4901337c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa4880ab000 tx=0x7fa4880191c0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 1 -- 172.21.15.5:0/3074800970 shutdown_connections 2024-09-13T11:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 1 --2- 172.21.15.5:0/3074800970 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa47c03b2a0 0x7fa47c03d760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 1 --2- 172.21.15.5:0/3074800970 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa490134810 0x7fa4901337c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 1 --2- 172.21.15.5:0/3074800970 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa49009dcc0 0x7fa490133280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 1 --2- 172.21.15.5:0/3074800970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa49009d1d0 0x7fa490132d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 1 -- 172.21.15.5:0/3074800970 >> 172.21.15.5:0/3074800970 conn(0x7fa4900a2930 msgr2=0x7fa490072a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 1 -- 172.21.15.5:0/3074800970 shutdown_connections 2024-09-13T11:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.143+0000 7fa498713640 1 -- 172.21.15.5:0/3074800970 wait complete. 2024-09-13T11:42:10.162 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-13T11:42:08.329057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167812,"kb_used_data":1464,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063228,"statfs":{"total":579820584960,"available":579648745472,"internally_reserved":0,"allocated":1499136,"data_stored":1545562,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003007"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.252094+0000","last_change":"2024-09-13T11:40:14.135514+0000","last_active":"2024-09-13T11:40:41.252094+0000","last_peered":"2024-09-13T11:40:41.252094+0000","last_clean":"2024-09-13T11:40:41.252094+0000","last_became_active":"2024-09-13T11:40:12.461041+0000","last_became_peered":"2024-09-13T11:40:12.461041+0000","last_unstale":"2024-09-13T11:40:41.252094+0000","last_undegraded":"2024-09-13T11:40:41.252094+0000","last_fullsized":"2024-09-13T11:40:41.252094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:50.409094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126878,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.466426+0000","last_change":"2024-09-13T11:40:14.143285+0000","last_active":"2024-09-13T11:40:41.466426+0000","last_peered":"2024-09-13T11:40:41.466426+0000","last_clean":"2024-09-13T11:40:41.466426+0000","last_became_active":"2024-09-13T11:40:12.106972+0000","last_became_peered":"2024-09-13T11:40:12.106972+0000","last_unstale":"2024-09-13T11:40:41.466426+0000","last_undegraded":"2024-09-13T11:40:41.466426+0000","last_fullsized":"2024-09-13T11:40:41.466426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:29.565401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001297091,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.252642+0000","last_change":"2024-09-13T11:40:14.135429+0000","last_active":"2024-09-13T11:40:41.252642+0000","last_peered":"2024-09-13T11:40:41.252642+0000","last_clean":"2024-09-13T11:40:41.252642+0000","last_became_active":"2024-09-13T11:40:12.120816+0000","last_became_peered":"2024-09-13T11:40:12.120816+0000","last_unstale":"2024-09-13T11:40:41.252642+0000","last_undegraded":"2024-09-13T11:40:41.252642+0000","last_fullsized":"2024-09-13T11:40:41.252642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:42.012635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011926969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.466384+0000","last_change":"2024-09-13T11:40:14.143208+0000","last_active":"2024-09-13T11:40:41.466384+0000","last_peered":"2024-09-13T11:40:41.466384+0000","last_clean":"2024-09-13T11:40:41.466384+0000","last_became_active":"2024-09-13T11:40:12.120092+0000","last_became_peered":"2024-09-13T11:40:12.120092+0000","last_unstale":"2024-09-13T11:40:41.466384+0000","last_undegraded":"2024-09-13T11:40:41.466384+0000","last_fullsized":"2024-09-13T11:40:41.466384+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:10.230543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001217161,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":36,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.461936+0000","last_change":"2024-09-13T11:40:14.139720+0000","last_active":"2024-09-13T11:40:41.461936+0000","last_peered":"2024-09-13T11:40:41.461936+0000","last_clean":"2024-09-13T11:40:41.461936+0000","last_became_active":"2024-09-13T11:40:12.110703+0000","last_became_peered":"2024-09-13T11:40:12.110703+0000","last_unstale":"2024-09-13T11:40:41.461936+0000","last_undegraded":"2024-09-13T11:40:41.461936+0000","last_fullsized":"2024-09-13T11:40:41.461936+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:57.779487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021711669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.250163+0000","last_change":"2024-09-13T11:40:14.131038+0000","last_active":"2024-09-13T11:40:41.250163+0000","last_peered":"2024-09-13T11:40:41.250163+0000","last_clean":"2024-09-13T11:40:41.250163+0000","last_became_active":"2024-09-13T11:40:12.102142+0000","last_became_peered":"2024-09-13T11:40:12.102142+0000","last_unstale":"2024-09-13T11:40:41.250163+0000","last_undegraded":"2024-09-13T11:40:41.250163+0000","last_fullsized":"2024-09-13T11:40:41.250163+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:15.679795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046429,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.251587+0000","last_change":"2024-09-13T11:40:14.135976+0000","last_active":"2024-09-13T11:40:41.251587+0000","last_peered":"2024-09-13T11:40:41.251587+0000","last_clean":"2024-09-13T11:40:41.251587+0000","last_became_active":"2024-09-13T11:40:12.105280+0000","last_became_peered":"2024-09-13T11:40:12.105280+0000","last_unstale":"2024-09-13T11:40:41.251587+0000","last_undegraded":"2024-09-13T11:40:41.251587+0000","last_fullsized":"2024-09-13T11:40:41.251587+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:20.791570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001476088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.462629+0000","last_change":"2024-09-13T11:40:14.138397+0000","last_active":"2024-09-13T11:40:41.462629+0000","last_peered":"2024-09-13T11:40:41.462629+0000","last_clean":"2024-09-13T11:40:41.462629+0000","last_became_active":"2024-09-13T11:40:12.113752+0000","last_became_peered":"2024-09-13T11:40:12.113752+0000","last_unstale":"2024-09-13T11:40:41.462629+0000","last_undegraded":"2024-09-13T11:40:41.462629+0000","last_fullsized":"2024-09-13T11:40:41.462629+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:50:52.508777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086960399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":125,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.251870+0000","last_change":"2024-09-13T11:40:05.394873+0000","last_active":"2024-09-13T11:40:41.251870+0000","last_peered":"2024-09-13T11:40:41.251870+0000","last_clean":"2024-09-13T11:40:41.251870+0000","last_became_active":"2024-09-13T11:40:05.394176+0000","last_became_peered":"2024-09-13T11:40:05.394176+0000","last_unstale":"2024-09-13T11:40:41.251870+0000","last_undegraded":"2024-09-13T11:40:41.251870+0000","last_fullsized":"2024-09-13T11:40:41.251870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_clean_scrub_stamp":"2024-09-13T11:40:04.377013+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:00:31.498589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607579,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":81297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607579,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":229376,"data_stored":81278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607579,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":286720,"data_stored":610206,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607579,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":81297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640284,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":229376,"data_stored":81278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640284,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":610206,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:42:10.163 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T11:42:10.163 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:42:10.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.275+0000 7f858be1a640 10 monclient: get_monmap_and_config 2024-09-13T11:42:10.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.275+0000 7f858be1a640 10 monclient: build_initial_monmap 2024-09-13T11:42:10.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.275+0000 7f858be1a640 10 monclient: monmap: 2024-09-13T11:42:10.281 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:10.281 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:10.281 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:10.280583+0000 2024-09-13T11:42:10.281 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:10.280583+0000 2024-09-13T11:42:10.281 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:10.281 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:10.281 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:10.281 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:10.281 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:10.282 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:10.282 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:10.282 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 1 Processor -- start 2024-09-13T11:42:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 1 -- start start 2024-09-13T11:42:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 10 monclient: init 2024-09-13T11:42:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-13T11:42:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85840a0ea0 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f85840a1ae0 0x7f858413a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 10 monclient(hunting): picked mon.noname-c con 0x7f85840a1ae0 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f858413ad30 0x7f858413d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 10 monclient(hunting): picked mon.noname-b con 0x7f858413ad30 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f858406f010 con 0x7f85840a0ea0 2024-09-13T11:42:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f858406f640 con 0x7f85840a1ae0 2024-09-13T11:42:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f85840a0000 con 0x7f858413ad30 2024-09-13T11:42:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858be1a640 10 monclient(hunting): authenticate will time out at 1133.629534s 2024-09-13T11:42:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f858413ad30 0x7f858413d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f858413ad30 0x7f858413d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:54938/0 (socket says 172.21.15.5:54938) 2024-09-13T11:42:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 1 -- 172.21.15.5:0/2373905335 learned_addr learned my addr 172.21.15.5:0/2373905335 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 10 monclient(hunting): get_auth_request con 0x7f858413ad30 auth_method 0 2024-09-13T11:42:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f85893b5640 1 --2- 172.21.15.5:0/2373905335 >> v2:172.21.15.39:3300/0 conn(0x7f85840a1ae0 0x7f858413a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f85893b5640 10 monclient(hunting): get_auth_request con 0x7f85840a1ae0 auth_method 0 2024-09-13T11:42:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f85893b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f85893b5640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f85893b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 10 monclient(hunting): handle_auth_done global_id 4448 payload 274 2024-09-13T11:42:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 1 -- 172.21.15.5:0/2373905335 >> v2:172.21.15.39:3300/0 conn(0x7f85840a1ae0 msgr2=0x7f858413a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 1 --2- 172.21.15.5:0/2373905335 >> v2:172.21.15.39:3300/0 conn(0x7f85840a1ae0 0x7f858413a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 1 -- 172.21.15.5:0/2373905335 >> v1:172.21.15.39:6789/0 conn(0x7f85840a0ea0 legacy=0x7f85840a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 10 monclient: _finish_hunting 0 2024-09-13T11:42:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 1 monclient: found mon.noname-b 2024-09-13T11:42:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:42:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.279+0000 7f858a3b7640 1 -- 172.21.15.5:0/2373905335 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858409fd80 con 0x7f858413ad30 2024-09-13T11:42:10.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 1 --2- 172.21.15.5:0/2373905335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f858413ad30 0x7f858413d120 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f857c009e50 tx=0x7f857c01c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d3e93c559b61d6c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:10.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 1 -- 172.21.15.5:0/2373905335 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c0a57a0 con 0x7f858413ad30 2024-09-13T11:42:10.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:10.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:42:10.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 10 monclient: dump: 2024-09-13T11:42:10.294 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:10.294 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:10.294 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.294 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.294 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:10.294 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:10.294 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:10.294 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:10.294 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:10.294 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:10.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 10 monclient: _finish_auth 0 2024-09-13T11:42:10.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 10 monclient: _check_auth_tickets 2024-09-13T11:42:10.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:10.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 1 -- 172.21.15.5:0/2373905335 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f857c0a5940 con 0x7f858413ad30 2024-09-13T11:42:10.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 5 monclient: authenticate success, global_id 4448 2024-09-13T11:42:10.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:10.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:10.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:10.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 4 set_mon_vals no callback set 2024-09-13T11:42:10.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 1 -- 172.21.15.5:0/2373905335 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c0a5d10 con 0x7f858413ad30 2024-09-13T11:42:10.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:10.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:10.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8588bb4640 10 monclient: dump: 2024-09-13T11:42:10.296 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:10.296 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:10.296 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.296 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.296 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:10.296 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:10.296 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:10.296 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:10.296 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:10.296 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:10.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient: shutdown 2024-09-13T11:42:10.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:10.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 -- 172.21.15.5:0/2373905335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f858413ad30 msgr2=0x7f858413d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 --2- 172.21.15.5:0/2373905335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f858413ad30 0x7f858413d120 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f857c009e50 tx=0x7f857c01c8c0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 -- 172.21.15.5:0/2373905335 shutdown_connections 2024-09-13T11:42:10.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 --2- 172.21.15.5:0/2373905335 >> v2:172.21.15.39:3300/0 conn(0x7f85840a1ae0 0x7f858413a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 --2- 172.21.15.5:0/2373905335 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f858413ad30 0x7f858413d120 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 -- 172.21.15.5:0/2373905335 >> 172.21.15.5:0/2373905335 conn(0x7f85840a2930 msgr2=0x7f85840a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:10.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 -- 172.21.15.5:0/2373905335 shutdown_connections 2024-09-13T11:42:10.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 -- 172.21.15.5:0/2373905335 wait complete. 2024-09-13T11:42:10.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient: build_initial_monmap 2024-09-13T11:42:10.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient: monmap: 2024-09-13T11:42:10.298 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:10.298 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:10.298 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:10.298 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:10.298 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:10.298 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:10.298 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:10.298 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:10.298 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:10.298 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:10.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 Processor -- start 2024-09-13T11:42:10.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 -- start start 2024-09-13T11:42:10.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient: init 2024-09-13T11:42:10.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:10.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:42:10.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85840a0ea0 0x7f8584131f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient(hunting): picked mon.noname-c con 0x7f85840a0ea0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:10.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85840a1ae0 0x7f8584132470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient(hunting): picked mon.noname-b con 0x7f85840a1ae0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:10.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f858413ad30 0x7f8584070900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient(hunting): picked mon.noname-a con 0x7f858413ad30 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:10.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:10.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8584140000 con 0x7f858413ad30 2024-09-13T11:42:10.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:10.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f858413fe80 con 0x7f85840a1ae0 2024-09-13T11:42:10.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:10.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f8584140180 con 0x7f85840a0ea0 2024-09-13T11:42:10.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:10.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient(hunting): authenticate will time out at 1133.634598s 2024-09-13T11:42:10.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f858413ad30 0x7f8584070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f858413ad30 0x7f8584070900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42920/0 (socket says 172.21.15.5:42920) 2024-09-13T11:42:10.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 1 -- 172.21.15.5:0/688283303 learned_addr learned my addr 172.21.15.5:0/688283303 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:10.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 10 monclient(hunting): get_auth_request con 0x7f858413ad30 auth_method 0 2024-09-13T11:42:10.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:10.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:10.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:10.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8589bb6640 1 --2- 172.21.15.5:0/688283303 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85840a0ea0 0x7f8584131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:10.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:10.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:10.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8589bb6640 10 monclient(hunting): get_auth_request con 0x7f85840a0ea0 auth_method 0 2024-09-13T11:42:10.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8589bb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:10.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8589bb6640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:10.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8589bb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:10.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f85893b5640 1 --2- 172.21.15.5:0/688283303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85840a1ae0 0x7f8584132470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f85893b5640 10 monclient(hunting): get_auth_request con 0x7f85840a1ae0 auth_method 0 2024-09-13T11:42:10.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f85893b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:10.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f85893b5640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:10.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f85893b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:10.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 10 monclient(hunting): handle_auth_done global_id 4605 payload 824 2024-09-13T11:42:10.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 1 -- 172.21.15.5:0/688283303 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85840a0ea0 msgr2=0x7f8584131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 1 --2- 172.21.15.5:0/688283303 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85840a0ea0 0x7f8584131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 1 -- 172.21.15.5:0/688283303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85840a1ae0 msgr2=0x7f8584132470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 1 --2- 172.21.15.5:0/688283303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85840a1ae0 0x7f8584132470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 10 monclient: _finish_hunting 0 2024-09-13T11:42:10.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 1 monclient: found mon.noname-a 2024-09-13T11:42:10.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:10.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:10.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 1 -- 172.21.15.5:0/688283303 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858411cac0 con 0x7f858413ad30 2024-09-13T11:42:10.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f8589bb6640 1 --2- 172.21.15.5:0/688283303 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85840a0ea0 0x7f8584131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:42:10.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858a3b7640 1 --2- 172.21.15.5:0/688283303 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f858413ad30 0x7f8584070900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f857c01a4f0 tx=0x7f857c019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:10.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 1 -- 172.21.15.5:0/688283303 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c0bc020 con 0x7f858413ad30 2024-09-13T11:42:10.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 10 monclient: dump: 2024-09-13T11:42:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:10.304 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 10 monclient: _finish_auth 0 2024-09-13T11:42:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:42:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 5 monclient: authenticate success, global_id 4605 2024-09-13T11:42:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 1 -- 172.21.15.5:0/688283303 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f857c0b8690 con 0x7f858413ad30 2024-09-13T11:42:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient: _renew_subs 2024-09-13T11:42:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 -- 172.21.15.5:0/688283303 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8584072c50 con 0x7f858413ad30 2024-09-13T11:42:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 1 -- 172.21.15.5:0/688283303 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c0b8a80 con 0x7f858413ad30 2024-09-13T11:42:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857a7fc640 10 monclient: dump: 2024-09-13T11:42:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:10.306 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:10.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient: _renew_subs 2024-09-13T11:42:10.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:10.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f857bfff640 4 set_mon_vals no callback set 2024-09-13T11:42:10.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.283+0000 7f858be1a640 1 -- 172.21.15.5:0/688283303 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f858413b3e0 con 0x7f858413ad30 2024-09-13T11:42:10.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.287+0000 7f858be1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:10.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.287+0000 7f858be1a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:10.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.287+0000 7f858be1a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:10.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.287+0000 7f858be1a640 1 -- 172.21.15.5:0/688283303 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85840a1280 con 0x7f858413ad30 2024-09-13T11:42:10.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.287+0000 7f857a7fc640 1 -- 172.21.15.5:0/688283303 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f857c0bf020 con 0x7f858413ad30 2024-09-13T11:42:10.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.287+0000 7f857a7fc640 1 --2- 172.21.15.5:0/688283303 >> v2:172.21.15.39:6812/1340838781 conn(0x7f856803b2a0 0x7f856803d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.287+0000 7f857a7fc640 1 -- 172.21.15.5:0/688283303 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f857c0ea1d0 con 0x7f858413ad30 2024-09-13T11:42:10.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.287+0000 7f8589bb6640 1 --2- 172.21.15.5:0/688283303 >> v2:172.21.15.39:6812/1340838781 conn(0x7f856803b2a0 0x7f856803d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.287+0000 7f8589bb6640 10 monclient: get_auth_request con 0x7f856803b2a0 auth_method 0 2024-09-13T11:42:10.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.287+0000 7f8589bb6640 1 --2- 172.21.15.5:0/688283303 >> v2:172.21.15.39:6812/1340838781 conn(0x7f856803b2a0 0x7f856803d760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8574000f70 tx=0x7f8574009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:10.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.291+0000 7f857a7fc640 1 -- 172.21.15.5:0/688283303 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f857c016600 con 0x7f858413ad30 2024-09-13T11:42:10.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.291+0000 7f857a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:10.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.291+0000 7f857a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.455+0000 7f858be1a640 1 -- 172.21.15.5:0/688283303 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f858413a980 con 0x7f856803b2a0 2024-09-13T11:42:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f857a7fc640 1 -- 172.21.15.5:0/688283303 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33921 (secure 0 0 0) 0x7f858413a980 con 0x7f856803b2a0 2024-09-13T11:42:10.462 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:42:10.463 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T11:42:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 1 -- 172.21.15.5:0/688283303 >> v2:172.21.15.39:6812/1340838781 conn(0x7f856803b2a0 msgr2=0x7f856803d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 1 --2- 172.21.15.5:0/688283303 >> v2:172.21.15.39:6812/1340838781 conn(0x7f856803b2a0 0x7f856803d760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8574000f70 tx=0x7f8574009040 comp rx=0 tx=0).stop 2024-09-13T11:42:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 10 monclient: shutdown 2024-09-13T11:42:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 1 -- 172.21.15.5:0/688283303 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f858413ad30 msgr2=0x7f8584070900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 1 --2- 172.21.15.5:0/688283303 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f858413ad30 0x7f8584070900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f857c01a4f0 tx=0x7f857c019660 comp rx=0 tx=0).stop 2024-09-13T11:42:10.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 1 -- 172.21.15.5:0/688283303 shutdown_connections 2024-09-13T11:42:10.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 1 --2- 172.21.15.5:0/688283303 >> v2:172.21.15.39:6812/1340838781 conn(0x7f856803b2a0 0x7f856803d760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 1 --2- 172.21.15.5:0/688283303 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f858413ad30 0x7f8584070900 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 1 --2- 172.21.15.5:0/688283303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85840a1ae0 0x7f8584132470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 1 --2- 172.21.15.5:0/688283303 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f85840a0ea0 0x7f8584131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 1 -- 172.21.15.5:0/688283303 >> 172.21.15.5:0/688283303 conn(0x7f85840a2930 msgr2=0x7f8584135640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:10.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 1 -- 172.21.15.5:0/688283303 shutdown_connections 2024-09-13T11:42:10.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.459+0000 7f858be1a640 1 -- 172.21.15.5:0/688283303 wait complete. 2024-09-13T11:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-13T11:42:10.329629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167812,"kb_used_data":1464,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063228,"statfs":{"total":579820584960,"available":579648745472,"internally_reserved":0,"allocated":1499136,"data_stored":1545562,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003033"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.252094+0000","last_change":"2024-09-13T11:40:14.135514+0000","last_active":"2024-09-13T11:40:41.252094+0000","last_peered":"2024-09-13T11:40:41.252094+0000","last_clean":"2024-09-13T11:40:41.252094+0000","last_became_active":"2024-09-13T11:40:12.461041+0000","last_became_peered":"2024-09-13T11:40:12.461041+0000","last_unstale":"2024-09-13T11:40:41.252094+0000","last_undegraded":"2024-09-13T11:40:41.252094+0000","last_fullsized":"2024-09-13T11:40:41.252094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:50.409094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126878,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.466426+0000","last_change":"2024-09-13T11:40:14.143285+0000","last_active":"2024-09-13T11:40:41.466426+0000","last_peered":"2024-09-13T11:40:41.466426+0000","last_clean":"2024-09-13T11:40:41.466426+0000","last_became_active":"2024-09-13T11:40:12.106972+0000","last_became_peered":"2024-09-13T11:40:12.106972+0000","last_unstale":"2024-09-13T11:40:41.466426+0000","last_undegraded":"2024-09-13T11:40:41.466426+0000","last_fullsized":"2024-09-13T11:40:41.466426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:29.565401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001297091,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.252642+0000","last_change":"2024-09-13T11:40:14.135429+0000","last_active":"2024-09-13T11:40:41.252642+0000","last_peered":"2024-09-13T11:40:41.252642+0000","last_clean":"2024-09-13T11:40:41.252642+0000","last_became_active":"2024-09-13T11:40:12.120816+0000","last_became_peered":"2024-09-13T11:40:12.120816+0000","last_unstale":"2024-09-13T11:40:41.252642+0000","last_undegraded":"2024-09-13T11:40:41.252642+0000","last_fullsized":"2024-09-13T11:40:41.252642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:42.012635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011926969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.466384+0000","last_change":"2024-09-13T11:40:14.143208+0000","last_active":"2024-09-13T11:40:41.466384+0000","last_peered":"2024-09-13T11:40:41.466384+0000","last_clean":"2024-09-13T11:40:41.466384+0000","last_became_active":"2024-09-13T11:40:12.120092+0000","last_became_peered":"2024-09-13T11:40:12.120092+0000","last_unstale":"2024-09-13T11:40:41.466384+0000","last_undegraded":"2024-09-13T11:40:41.466384+0000","last_fullsized":"2024-09-13T11:40:41.466384+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:10.230543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001217161,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":36,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.461936+0000","last_change":"2024-09-13T11:40:14.139720+0000","last_active":"2024-09-13T11:40:41.461936+0000","last_peered":"2024-09-13T11:40:41.461936+0000","last_clean":"2024-09-13T11:40:41.461936+0000","last_became_active":"2024-09-13T11:40:12.110703+0000","last_became_peered":"2024-09-13T11:40:12.110703+0000","last_unstale":"2024-09-13T11:40:41.461936+0000","last_undegraded":"2024-09-13T11:40:41.461936+0000","last_fullsized":"2024-09-13T11:40:41.461936+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:57.779487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021711669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.250163+0000","last_change":"2024-09-13T11:40:14.131038+0000","last_active":"2024-09-13T11:40:41.250163+0000","last_peered":"2024-09-13T11:40:41.250163+0000","last_clean":"2024-09-13T11:40:41.250163+0000","last_became_active":"2024-09-13T11:40:12.102142+0000","last_became_peered":"2024-09-13T11:40:12.102142+0000","last_unstale":"2024-09-13T11:40:41.250163+0000","last_undegraded":"2024-09-13T11:40:41.250163+0000","last_fullsized":"2024-09-13T11:40:41.250163+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:15.679795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046429,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.251587+0000","last_change":"2024-09-13T11:40:14.135976+0000","last_active":"2024-09-13T11:40:41.251587+0000","last_peered":"2024-09-13T11:40:41.251587+0000","last_clean":"2024-09-13T11:40:41.251587+0000","last_became_active":"2024-09-13T11:40:12.105280+0000","last_became_peered":"2024-09-13T11:40:12.105280+0000","last_unstale":"2024-09-13T11:40:41.251587+0000","last_undegraded":"2024-09-13T11:40:41.251587+0000","last_fullsized":"2024-09-13T11:40:41.251587+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:20.791570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001476088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.462629+0000","last_change":"2024-09-13T11:40:14.138397+0000","last_active":"2024-09-13T11:40:41.462629+0000","last_peered":"2024-09-13T11:40:41.462629+0000","last_clean":"2024-09-13T11:40:41.462629+0000","last_became_active":"2024-09-13T11:40:12.113752+0000","last_became_peered":"2024-09-13T11:40:12.113752+0000","last_unstale":"2024-09-13T11:40:41.462629+0000","last_undegraded":"2024-09-13T11:40:41.462629+0000","last_fullsized":"2024-09-13T11:40:41.462629+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:50:52.508777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086960399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":126,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.252004+0000","last_change":"2024-09-13T11:40:05.394873+0000","last_active":"2024-09-13T11:40:41.252004+0000","last_peered":"2024-09-13T11:40:41.252004+0000","last_clean":"2024-09-13T11:40:41.252004+0000","last_became_active":"2024-09-13T11:40:05.394176+0000","last_became_peered":"2024-09-13T11:40:05.394176+0000","last_unstale":"2024-09-13T11:40:41.252004+0000","last_undegraded":"2024-09-13T11:40:41.252004+0000","last_fullsized":"2024-09-13T11:40:41.252004+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_clean_scrub_stamp":"2024-09-13T11:40:04.377013+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:00:31.498589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607580,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":81297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607580,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":229376,"data_stored":81278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607580,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":286720,"data_stored":610206,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607580,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":81297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640284,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":229376,"data_stored":81278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640284,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":610206,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:42:10.481 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:42:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient: get_monmap_and_config 2024-09-13T11:42:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient: build_initial_monmap 2024-09-13T11:42:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient: monmap: 2024-09-13T11:42:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:10.599173+0000 2024-09-13T11:42:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:10.599173+0000 2024-09-13T11:42:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:10.600 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 1 Processor -- start 2024-09-13T11:42:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 1 -- start start 2024-09-13T11:42:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient: init 2024-09-13T11:42:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-13T11:42:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f95080a0ea0 0x7f95080a1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient(hunting): picked mon.noname-a con 0x7f95080a0ea0 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f95080a1ae0 0x7f95081354b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient(hunting): picked mon.noname-c con 0x7f95080a1ae0 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 0x7f9508135e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient(hunting): picked mon.noname-b con 0x7f95081359f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f950806f640 con 0x7f95080a0ea0 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f95080a0000 con 0x7f95080a1ae0 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f950806f010 con 0x7f95081359f0 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.595+0000 7f950edde640 10 monclient(hunting): authenticate will time out at 1133.947916s 2024-09-13T11:42:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f95080a0ea0 0x7f95080a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950d37b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 0x7f9508135e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f95080a0ea0 0x7f95080a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42930/0 (socket says 172.21.15.5:42930) 2024-09-13T11:42:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950d37b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 0x7f9508135e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:54968/0 (socket says 172.21.15.5:54968) 2024-09-13T11:42:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 1 -- 172.21.15.5:0/2057407653 learned_addr learned my addr 172.21.15.5:0/2057407653 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 10 monclient(hunting): get_auth_request con 0x7f95080a0ea0 auth_method 0 2024-09-13T11:42:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950d37b640 10 monclient(hunting): get_auth_request con 0x7f95081359f0 auth_method 0 2024-09-13T11:42:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950d37b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950d37b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950d37b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f9507fff640 1 --2- 172.21.15.5:0/2057407653 >> v2:172.21.15.39:3300/0 conn(0x7f95080a1ae0 0x7f95081354b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f9507fff640 10 monclient(hunting): get_auth_request con 0x7f95080a1ae0 auth_method 0 2024-09-13T11:42:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f9507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f9507fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f9507fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950d37b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950d37b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950d37b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f9507fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f9507fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f9507fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 10 monclient(hunting): handle_auth_done global_id 4608 payload 274 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 1 -- 172.21.15.5:0/2057407653 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 msgr2=0x7f9508135e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 1 --2- 172.21.15.5:0/2057407653 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 0x7f9508135e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 1 -- 172.21.15.5:0/2057407653 >> v2:172.21.15.39:3300/0 conn(0x7f95080a1ae0 msgr2=0x7f95081354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 1 --2- 172.21.15.5:0/2057407653 >> v2:172.21.15.39:3300/0 conn(0x7f95080a1ae0 0x7f95081354b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 10 monclient: _finish_hunting 0 2024-09-13T11:42:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 1 monclient: found mon.noname-a 2024-09-13T11:42:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 1 -- 172.21.15.5:0/2057407653 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950809fd80 con 0x7f95080a0ea0 2024-09-13T11:42:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950d37b640 1 --2- 172.21.15.5:0/2057407653 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 0x7f9508135e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950cb7a640 1 --2- 172.21.15.5:0/2057407653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95080a0ea0 0x7f95080a1280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f950806efa0 tx=0x7f94fc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e04d3a705df8b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 1 -- 172.21.15.5:0/2057407653 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc0b0070 con 0x7f95080a0ea0 2024-09-13T11:42:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 10 monclient: dump: 2024-09-13T11:42:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:10.610 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 10 monclient: _finish_auth 0 2024-09-13T11:42:10.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 10 monclient: _check_auth_tickets 2024-09-13T11:42:10.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:10.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950edde640 5 monclient: authenticate success, global_id 4608 2024-09-13T11:42:10.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 1 -- 172.21.15.5:0/2057407653 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94fc0a4e50 con 0x7f95080a0ea0 2024-09-13T11:42:10.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950edde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:10.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:10.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950edde640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:10.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950edde640 4 set_mon_vals no callback set 2024-09-13T11:42:10.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 1 -- 172.21.15.5:0/2057407653 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc0ab820 con 0x7f95080a0ea0 2024-09-13T11:42:10.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:10.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:10.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f95077fe640 10 monclient: dump: 2024-09-13T11:42:10.612 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:10.612 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:10.612 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.612 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.612 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:10.612 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:10.613 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:10.613 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:10.613 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:10.613 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:10.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950edde640 10 monclient: shutdown 2024-09-13T11:42:10.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950edde640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:10.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950edde640 1 -- 172.21.15.5:0/2057407653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95080a0ea0 msgr2=0x7f95080a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.599+0000 7f950edde640 1 --2- 172.21.15.5:0/2057407653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95080a0ea0 0x7f95080a1280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f950806efa0 tx=0x7f94fc01c3e0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 -- 172.21.15.5:0/2057407653 shutdown_connections 2024-09-13T11:42:10.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 --2- 172.21.15.5:0/2057407653 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 0x7f9508135e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 --2- 172.21.15.5:0/2057407653 >> v2:172.21.15.39:3300/0 conn(0x7f95080a1ae0 0x7f95081354b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 --2- 172.21.15.5:0/2057407653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95080a0ea0 0x7f95080a1280 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 -- 172.21.15.5:0/2057407653 >> 172.21.15.5:0/2057407653 conn(0x7f95080a2930 msgr2=0x7f95080a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:10.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 -- 172.21.15.5:0/2057407653 shutdown_connections 2024-09-13T11:42:10.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 -- 172.21.15.5:0/2057407653 wait complete. 2024-09-13T11:42:10.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient: build_initial_monmap 2024-09-13T11:42:10.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient: monmap: 2024-09-13T11:42:10.622 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:10.622 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:10.623 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:10.623 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:10.623 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:10.623 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:10.623 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:10.623 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:10.623 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:10.624 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:10.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 Processor -- start 2024-09-13T11:42:10.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 -- start start 2024-09-13T11:42:10.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient: init 2024-09-13T11:42:10.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:10.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:42:10.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95080a0ea0 0x7f950811c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient(hunting): picked mon.noname-b con 0x7f95080a0ea0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:10.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95080a1ae0 0x7f950813ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient(hunting): picked mon.noname-a con 0x7f95080a1ae0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:10.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 0x7f9508132600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient(hunting): picked mon.noname-c con 0x7f95081359f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:10.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:10.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f95080a0ca0 con 0x7f95080a1ae0 2024-09-13T11:42:10.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:10.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f95080a0b20 con 0x7f95080a0ea0 2024-09-13T11:42:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f950813f430 con 0x7f95081359f0 2024-09-13T11:42:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient(hunting): authenticate will time out at 1133.954068s 2024-09-13T11:42:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f9507fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95080a1ae0 0x7f950813ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f9507fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95080a1ae0 0x7f950813ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42946/0 (socket says 172.21.15.5:42946) 2024-09-13T11:42:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f9507fff640 1 -- 172.21.15.5:0/2571638056 learned_addr learned my addr 172.21.15.5:0/2571638056 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f9507fff640 10 monclient(hunting): get_auth_request con 0x7f95080a1ae0 auth_method 0 2024-09-13T11:42:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f9507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:10.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f9507fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:10.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f9507fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:10.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 1 --2- 172.21.15.5:0/2571638056 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 0x7f9508132600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 10 monclient(hunting): get_auth_request con 0x7f95081359f0 auth_method 0 2024-09-13T11:42:10.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:10.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:10.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:10.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f9507fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:10.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f9507fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:10.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f9507fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:10.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:10.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:10.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:10.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950cb7a640 1 --2- 172.21.15.5:0/2571638056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95080a0ea0 0x7f950811c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950cb7a640 10 monclient(hunting): get_auth_request con 0x7f95080a0ea0 auth_method 0 2024-09-13T11:42:10.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950cb7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:10.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950cb7a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:10.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950cb7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:10.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 10 monclient(hunting): handle_auth_done global_id 4457 payload 824 2024-09-13T11:42:10.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 1 -- 172.21.15.5:0/2571638056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95080a0ea0 msgr2=0x7f950811c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 1 --2- 172.21.15.5:0/2571638056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95080a0ea0 0x7f950811c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 1 -- 172.21.15.5:0/2571638056 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95080a1ae0 msgr2=0x7f950813ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 1 --2- 172.21.15.5:0/2571638056 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95080a1ae0 0x7f950813ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 10 monclient: _finish_hunting 0 2024-09-13T11:42:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 1 monclient: found mon.noname-c 2024-09-13T11:42:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:42:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 1 -- 172.21.15.5:0/2571638056 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950806dbf0 con 0x7f95081359f0 2024-09-13T11:42:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f9507fff640 1 --2- 172.21.15.5:0/2571638056 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95080a1ae0 0x7f950813ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950d37b640 1 --2- 172.21.15.5:0/2571638056 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 0x7f9508132600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f950000efc0 tx=0x7f950000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 1 -- 172.21.15.5:0/2571638056 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500018030 con 0x7f95081359f0 2024-09-13T11:42:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:42:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 10 monclient: dump: 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 10 monclient: _finish_auth 0 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 10 monclient: _check_auth_tickets 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 1 -- 172.21.15.5:0/2571638056 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f950000ccd0 con 0x7f95081359f0 2024-09-13T11:42:10.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:10.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 1 -- 172.21.15.5:0/2571638056 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500003bc0 con 0x7f95081359f0 2024-09-13T11:42:10.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:10.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:10.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f95057fa640 10 monclient: dump: 2024-09-13T11:42:10.632 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:10.632 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:10.632 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.632 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:10.632 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:10.633 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:10.633 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:10.633 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:10.633 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:10.633 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:10.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 5 monclient: authenticate success, global_id 4457 2024-09-13T11:42:10.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient: _renew_subs 2024-09-13T11:42:10.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:10.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f9506ffd640 4 set_mon_vals no callback set 2024-09-13T11:42:10.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 -- 172.21.15.5:0/2571638056 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950809c390 con 0x7f95081359f0 2024-09-13T11:42:10.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient: _renew_subs 2024-09-13T11:42:10.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:10.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 -- 172.21.15.5:0/2571638056 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f950811dfc0 con 0x7f95081359f0 2024-09-13T11:42:10.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:10.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.603+0000 7f950edde640 1 -- 172.21.15.5:0/2571638056 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95081354b0 con 0x7f95081359f0 2024-09-13T11:42:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.607+0000 7f95057fa640 1 -- 172.21.15.5:0/2571638056 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f950001b070 con 0x7f95081359f0 2024-09-13T11:42:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.607+0000 7f95057fa640 1 --2- 172.21.15.5:0/2571638056 >> v2:172.21.15.39:6812/1340838781 conn(0x7f94f403b2d0 0x7f94f403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.607+0000 7f95057fa640 1 -- 172.21.15.5:0/2571638056 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f950004f5c0 con 0x7f95081359f0 2024-09-13T11:42:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.607+0000 7f950cb7a640 1 --2- 172.21.15.5:0/2571638056 >> v2:172.21.15.39:6812/1340838781 conn(0x7f94f403b2d0 0x7f94f403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.607+0000 7f950cb7a640 10 monclient: get_auth_request con 0x7f94f403b2d0 auth_method 0 2024-09-13T11:42:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.607+0000 7f950cb7a640 1 --2- 172.21.15.5:0/2571638056 >> v2:172.21.15.39:6812/1340838781 conn(0x7f94f403b2d0 0x7f94f403d790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0ad000 tx=0x7f94fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.611+0000 7f95057fa640 1 -- 172.21.15.5:0/2571638056 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f9500022070 con 0x7f95081359f0 2024-09-13T11:42:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.611+0000 7f95057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.611+0000 7f95057fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:10.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.779+0000 7f950edde640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:42:10.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.779+0000 7f950edde640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:42:10.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.779+0000 7f950edde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:10.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.779+0000 7f950edde640 1 -- 172.21.15.5:0/2571638056 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9508069ff0 con 0x7f95081359f0 2024-09-13T11:42:10.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.783+0000 7f95057fa640 1 -- 172.21.15.5:0/2571638056 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+10829 (secure 0 0 0) 0x7f9500010040 con 0x7f95081359f0 2024-09-13T11:42:10.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.783+0000 7f95057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T11:42:10.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.783+0000 7f95057fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:42:10.786 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:42:10.787 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":26,"fsid":"59046471-79a9-4d90-9616-da0b446053a7","created":"2024-09-13T11:39:53.326128+0000","modified":"2024-09-13T11:40:41.238593+0000","last_up_change":"2024-09-13T11:40:03.374234+0000","last_in_change":"2024-09-13T11:39:55.104089+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:40:04.344231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:40:10.361569+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0803d65c-46ec-49b5-a45e-46bcf59ae3f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":4278099835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":4278099835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":4278099835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":4278099835}]},"public_addr":"172.21.15.5:6804/4278099835","cluster_addr":"172.21.15.5:6805/4278099835","heartbeat_back_addr":"172.21.15.5:6807/4278099835","heartbeat_front_addr":"172.21.15.5:6806/4278099835","state":["exists","up"]},{"osd":1,"uuid":"fd592a78-172c-4e1c-9f3e-c49e3e3304d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":1447734001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":1447734001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":1447734001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":1447734001}]},"public_addr":"172.21.15.5:6808/1447734001","cluster_addr":"172.21.15.5:6809/1447734001","heartbeat_back_addr":"172.21.15.5:6811/1447734001","heartbeat_front_addr":"172.21.15.5:6810/1447734001","state":["exists","up"]},{"osd":2,"uuid":"c3cec868-fd4e-472c-83b8-bc2e466671ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":1907498696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":1907498696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":1907498696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":1907498696}]},"public_addr":"172.21.15.5:6800/1907498696","cluster_addr":"172.21.15.5:6801/1907498696","heartbeat_back_addr":"172.21.15.5:6803/1907498696","heartbeat_front_addr":"172.21.15.5:6802/1907498696","state":["exists","up"]},{"osd":3,"uuid":"748f14f6-a1d6-4669-9f81-6b1cd7cd5a60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3188120659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":3188120659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":3188120659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":3188120659}]},"public_addr":"172.21.15.39:6804/3188120659","cluster_addr":"172.21.15.39:6805/3188120659","heartbeat_back_addr":"172.21.15.39:6807/3188120659","heartbeat_front_addr":"172.21.15.39:6806/3188120659","state":["exists","up"]},{"osd":4,"uuid":"f3a05376-7e83-46f9-b3e6-97b31a0942a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":4264535364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":4264535364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":4264535364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4264535364}]},"public_addr":"172.21.15.39:6800/4264535364","cluster_addr":"172.21.15.39:6801/4264535364","heartbeat_back_addr":"172.21.15.39:6803/4264535364","heartbeat_front_addr":"172.21.15.39:6802/4264535364","state":["exists","up"]},{"osd":5,"uuid":"e3b64b2d-c3e5-45d1-90a9-225892061800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1185247880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":1185247880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":1185247880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1185247880}]},"public_addr":"172.21.15.39:6808/1185247880","cluster_addr":"172.21.15.39:6809/1185247880","heartbeat_back_addr":"172.21.15.39:6811/1185247880","heartbeat_front_addr":"172.21.15.39:6810/1185247880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.244929+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.890770+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.206643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.846803+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:01.612582+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:40:02.321712+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.783+0000 7f950edde640 1 -- 172.21.15.5:0/2571638056 >> v2:172.21.15.39:6812/1340838781 conn(0x7f94f403b2d0 msgr2=0x7f94f403d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.783+0000 7f950edde640 1 --2- 172.21.15.5:0/2571638056 >> v2:172.21.15.39:6812/1340838781 conn(0x7f94f403b2d0 0x7f94f403d790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0ad000 tx=0x7f94fc008040 comp rx=0 tx=0).stop 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.783+0000 7f950edde640 10 monclient: shutdown 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.783+0000 7f950edde640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.783+0000 7f950edde640 1 -- 172.21.15.5:0/2571638056 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 msgr2=0x7f9508132600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.783+0000 7f950edde640 1 --2- 172.21.15.5:0/2571638056 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 0x7f9508132600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f950000efc0 tx=0x7f950000c490 comp rx=0 tx=0).stop 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.787+0000 7f950edde640 1 -- 172.21.15.5:0/2571638056 shutdown_connections 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.787+0000 7f950edde640 1 --2- 172.21.15.5:0/2571638056 >> v2:172.21.15.39:6812/1340838781 conn(0x7f94f403b2d0 0x7f94f403d790 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.787+0000 7f950edde640 1 --2- 172.21.15.5:0/2571638056 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f95081359f0 0x7f9508132600 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.787+0000 7f950edde640 1 --2- 172.21.15.5:0/2571638056 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f95080a1ae0 0x7f950813ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.787+0000 7f950edde640 1 --2- 172.21.15.5:0/2571638056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95080a0ea0 0x7f950811c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.787+0000 7f950edde640 1 -- 172.21.15.5:0/2571638056 >> 172.21.15.5:0/2571638056 conn(0x7f95080a2930 msgr2=0x7f95080a48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.787+0000 7f950edde640 1 -- 172.21.15.5:0/2571638056 shutdown_connections 2024-09-13T11:42:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:10.787+0000 7f950edde640 1 -- 172.21.15.5:0/2571638056 wait complete. 2024-09-13T11:42:11.806 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T11:42:11.807 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:42:11.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient: get_monmap_and_config 2024-09-13T11:42:11.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient: build_initial_monmap 2024-09-13T11:42:11.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient: monmap: 2024-09-13T11:42:11.947 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:11.947 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:11.947 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:11.946844+0000 2024-09-13T11:42:11.947 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:11.946844+0000 2024-09-13T11:42:11.947 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:11.947 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:11.947 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:11.948 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:11.948 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:11.948 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:11.948 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:11.948 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:11.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 1 Processor -- start 2024-09-13T11:42:11.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 1 -- start start 2024-09-13T11:42:11.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient: init 2024-09-13T11:42:11.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1228135650 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1228136140 0x7f122813ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient(hunting): picked mon.noname-c con 0x7f1228136140 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f122813b0c0 0x7f122813d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient(hunting): picked mon.noname-b con 0x7f122813b0c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12280a1da0 con 0x7f1228135650 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f122809fca0 con 0x7f1228136140 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f12280a2080 con 0x7f122813b0c0 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122d1b9640 10 monclient(hunting): authenticate will time out at 1135.295604s 2024-09-13T11:42:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122759e640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f122813b0c0 0x7f122813d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f1226d9d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1228135650 0x7f1228135a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60686/0 (socket says 172.21.15.5:60686) 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f1226d9d640 1 -- 172.21.15.5:0/3416765656 learned_addr learned my addr 172.21.15.5:0/3416765656 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122759e640 10 monclient(hunting): get_auth_request con 0x7f122813b0c0 auth_method 0 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122759e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.943+0000 7f122759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122659c640 1 --2- 172.21.15.5:0/3416765656 >> v2:172.21.15.39:3300/0 conn(0x7f1228136140 0x7f122813ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122659c640 10 monclient(hunting): get_auth_request con 0x7f1228136140 auth_method 0 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:11.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122759e640 10 monclient(hunting): handle_auth_done global_id 4460 payload 274 2024-09-13T11:42:11.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 1 -- 172.21.15.5:0/3416765656 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3422408016 0 0) 0x7f1218001120 con 0x7f1228135650 2024-09-13T11:42:11.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122759e640 1 -- 172.21.15.5:0/3416765656 >> v2:172.21.15.39:3300/0 conn(0x7f1228136140 msgr2=0x7f122813ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:11.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122759e640 1 --2- 172.21.15.5:0/3416765656 >> v2:172.21.15.39:3300/0 conn(0x7f1228136140 0x7f122813ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:11.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122759e640 1 -- 172.21.15.5:0/3416765656 >> v1:172.21.15.5:6789/0 conn(0x7f1228135650 legacy=0x7f1228135a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:11.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122759e640 10 monclient: _finish_hunting 0 2024-09-13T11:42:11.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122759e640 1 monclient: found mon.noname-b 2024-09-13T11:42:11.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:11.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:42:11.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122759e640 1 -- 172.21.15.5:0/3416765656 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12280a2360 con 0x7f122813b0c0 2024-09-13T11:42:11.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T11:42:11.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122759e640 1 --2- 172.21.15.5:0/3416765656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f122813b0c0 0x7f122813d4b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1220009d90 tx=0x7f122001c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ca33dfa8fe81d0e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:11.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 1 -- 172.21.15.5:0/3416765656 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12200b2070 con 0x7f122813b0c0 2024-09-13T11:42:11.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:11.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:42:11.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 10 monclient: dump: 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 10 monclient: _finish_auth 0 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 10 monclient: _check_auth_tickets 2024-09-13T11:42:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 1 -- 172.21.15.5:0/3416765656 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12200a5800 con 0x7f122813b0c0 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 5 monclient: authenticate success, global_id 4460 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 4 set_mon_vals no callback set 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 1 -- 172.21.15.5:0/3416765656 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12200a5bd0 con 0x7f122813b0c0 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f1225d9b640 10 monclient: dump: 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:11.962 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:11.962 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:11.962 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:11.962 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:11.962 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:11.962 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:11.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 10 monclient: shutdown 2024-09-13T11:42:11.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:11.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 1 -- 172.21.15.5:0/3416765656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f122813b0c0 msgr2=0x7f122813d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:11.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 1 --2- 172.21.15.5:0/3416765656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f122813b0c0 0x7f122813d4b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1220009d90 tx=0x7f122001c830 comp rx=0 tx=0).stop 2024-09-13T11:42:11.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 1 -- 172.21.15.5:0/3416765656 shutdown_connections 2024-09-13T11:42:11.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 1 --2- 172.21.15.5:0/3416765656 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f122813b0c0 0x7f122813d4b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 1 --2- 172.21.15.5:0/3416765656 >> v2:172.21.15.39:3300/0 conn(0x7f1228136140 0x7f122813ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 1 -- 172.21.15.5:0/3416765656 >> 172.21.15.5:0/3416765656 conn(0x7f12280a6780 msgr2=0x7f12280a2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 1 -- 172.21.15.5:0/3416765656 shutdown_connections 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 1 -- 172.21.15.5:0/3416765656 wait complete. 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 10 monclient: build_initial_monmap 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.947+0000 7f122d1b9640 10 monclient: monmap: 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:11.963 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 1 Processor -- start 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 1 -- start start 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient: init 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1228135650 0x7f122809d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient(hunting): picked mon.noname-b con 0x7f1228135650 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1228136140 0x7f122809d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient(hunting): picked mon.noname-a con 0x7f1228136140 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f122813b0c0 0x7f122809de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:11.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient(hunting): picked mon.noname-c con 0x7f122813b0c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1228140390 con 0x7f1228136140 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1228140210 con 0x7f1228135650 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f1228140510 con 0x7f122813b0c0 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient(hunting): authenticate will time out at 1135.300702s 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1228136140 0x7f122809d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1228136140 0x7f122809d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42954/0 (socket says 172.21.15.5:42954) 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 1 -- 172.21.15.5:0/1790134644 learned_addr learned my addr 172.21.15.5:0/1790134644 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 10 monclient(hunting): get_auth_request con 0x7f1228136140 auth_method 0 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:11.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122759e640 1 --2- 172.21.15.5:0/1790134644 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f122813b0c0 0x7f122809de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122759e640 10 monclient(hunting): get_auth_request con 0x7f122813b0c0 auth_method 0 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122759e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f1226d9d640 1 --2- 172.21.15.5:0/1790134644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1228135650 0x7f122809d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f1226d9d640 10 monclient(hunting): get_auth_request con 0x7f1228135650 auth_method 0 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f1226d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f1226d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f1226d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 10 monclient(hunting): handle_auth_done global_id 4617 payload 824 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 1 -- 172.21.15.5:0/1790134644 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f122813b0c0 msgr2=0x7f122809de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 1 --2- 172.21.15.5:0/1790134644 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f122813b0c0 0x7f122809de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 1 -- 172.21.15.5:0/1790134644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1228135650 msgr2=0x7f122809d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 1 --2- 172.21.15.5:0/1790134644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1228135650 0x7f122809d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 10 monclient: _finish_hunting 0 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 1 monclient: found mon.noname-a 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 1 -- 172.21.15.5:0/1790134644 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1228120ec0 con 0x7f1228136140 2024-09-13T11:42:11.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122759e640 1 --2- 172.21.15.5:0/1790134644 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f122813b0c0 0x7f122809de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122659c640 1 --2- 172.21.15.5:0/1790134644 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1228136140 0x7f122809d920 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f121c00d800 tx=0x7f121c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 1 -- 172.21.15.5:0/1790134644 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c00b6c0 con 0x7f1228136140 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 10 monclient: dump: 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:11.968 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 10 monclient: _finish_auth 0 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 1 -- 172.21.15.5:0/1790134644 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f121c019070 con 0x7f1228136140 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 1 -- 172.21.15.5:0/1790134644 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c00ba60 con 0x7f1228136140 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 10 monclient: dump: 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:11.969 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 5 monclient: authenticate success, global_id 4617 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient: _renew_subs 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122559a640 4 set_mon_vals no callback set 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 1 -- 172.21.15.5:0/1790134644 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12281211a0 con 0x7f1228136140 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient: _renew_subs 2024-09-13T11:42:11.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 1 -- 172.21.15.5:0/1790134644 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f122809d0d0 con 0x7f1228136140 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122d1b9640 1 -- 172.21.15.5:0/1790134644 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1228135a30 con 0x7f1228136140 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 1 -- 172.21.15.5:0/1790134644 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f121c00bc00 con 0x7f1228136140 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 1 --2- 172.21.15.5:0/1790134644 >> v2:172.21.15.39:6812/1340838781 conn(0x7f121403b2a0 0x7f121403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 1 -- 172.21.15.5:0/1790134644 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f121c04f090 con 0x7f1228136140 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 1 --2- 172.21.15.5:0/1790134644 >> v2:172.21.15.5:6804/4278099835 conn(0x7f1214040d40 0x7f1214043160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 1 -- 172.21.15.5:0/1790134644 --> v2:172.21.15.5:6804/4278099835 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12140437d0 con 0x7f1214040d40 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 1 -- 172.21.15.5:0/1790134644 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f121c056050 con 0x7f1228136140 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f120f7fe640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122759e640 1 --2- 172.21.15.5:0/1790134644 >> v2:172.21.15.5:6804/4278099835 conn(0x7f1214040d40 0x7f1214043160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122759e640 10 monclient: get_auth_request con 0x7f1214040d40 auth_method 0 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f1226d9d640 1 --2- 172.21.15.5:0/1790134644 >> v2:172.21.15.39:6812/1340838781 conn(0x7f121403b2a0 0x7f121403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:11.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f1226d9d640 10 monclient: get_auth_request con 0x7f121403b2a0 auth_method 0 2024-09-13T11:42:11.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.951+0000 7f122759e640 1 --2- 172.21.15.5:0/1790134644 >> v2:172.21.15.5:6804/4278099835 conn(0x7f1214040d40 0x7f1214043160 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:11.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.955+0000 7f1226d9d640 1 --2- 172.21.15.5:0/1790134644 >> v2:172.21.15.39:6812/1340838781 conn(0x7f121403b2a0 0x7f121403d760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1218007fd0 tx=0x7f12180079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:11.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.955+0000 7f120f7fe640 1 -- 172.21.15.5:0/1790134644 <== osd.0 v2:172.21.15.5:6804/4278099835 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f12140437d0 con 0x7f1214040d40 2024-09-13T11:42:11.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 1 -- 172.21.15.5:0/1790134644 --> v2:172.21.15.5:6804/4278099835 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f122813a400 con 0x7f1214040d40 2024-09-13T11:42:11.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f120f7fe640 1 -- 172.21.15.5:0/1790134644 <== osd.0 v2:172.21.15.5:6804/4278099835 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f122813a400 con 0x7f1214040d40 2024-09-13T11:42:11.973 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:42:11.973 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:42:11.973 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:42:11.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 1 -- 172.21.15.5:0/1790134644 >> v2:172.21.15.5:6804/4278099835 conn(0x7f1214040d40 msgr2=0x7f1214043160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 1 --2- 172.21.15.5:0/1790134644 >> v2:172.21.15.5:6804/4278099835 conn(0x7f1214040d40 0x7f1214043160 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 1 -- 172.21.15.5:0/1790134644 >> v2:172.21.15.39:6812/1340838781 conn(0x7f121403b2a0 msgr2=0x7f121403d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 1 --2- 172.21.15.5:0/1790134644 >> v2:172.21.15.39:6812/1340838781 conn(0x7f121403b2a0 0x7f121403d760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1218007fd0 tx=0x7f12180079c0 comp rx=0 tx=0).stop 2024-09-13T11:42:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 10 monclient: shutdown 2024-09-13T11:42:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 1 -- 172.21.15.5:0/1790134644 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1228136140 msgr2=0x7f122809d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 1 --2- 172.21.15.5:0/1790134644 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1228136140 0x7f122809d920 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f121c00d800 tx=0x7f121c00dcc0 comp rx=0 tx=0).stop 2024-09-13T11:42:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122759e640 1 -- 172.21.15.5:0/1790134644 reap_dead start 2024-09-13T11:42:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 1 -- 172.21.15.5:0/1790134644 shutdown_connections 2024-09-13T11:42:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 1 -- 172.21.15.5:0/1790134644 >> 172.21.15.5:0/1790134644 conn(0x7f12280a6780 msgr2=0x7f122813c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 1 -- 172.21.15.5:0/1790134644 shutdown_connections 2024-09-13T11:42:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:11.967+0000 7f122d1b9640 1 -- 172.21.15.5:0/1790134644 wait complete. 2024-09-13T11:42:11.987 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T11:42:12.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 10 monclient: get_monmap_and_config 2024-09-13T11:42:12.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 10 monclient: build_initial_monmap 2024-09-13T11:42:12.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 10 monclient: monmap: 2024-09-13T11:42:12.101 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:12.101 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.102 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:12.101365+0000 2024-09-13T11:42:12.102 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:12.101365+0000 2024-09-13T11:42:12.102 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:12.102 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.102 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:12.102 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:12.102 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:12.102 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:12.102 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:12.102 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 1 Processor -- start 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 1 -- start start 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 10 monclient: init 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f10fc134810 0x7f10fc134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 10 monclient(hunting): picked mon.noname-c con 0x7f10fc134810 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10fc09d1d0 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10fc09dcf0 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10fc09ce60 con 0x7f10fc09d1d0 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10fc09eeb0 con 0x7f10fc09dcf0 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f10fc0a0950 con 0x7f10fc134810 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:12.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 10 monclient(hunting): authenticate will time out at 1135.450735s 2024-09-13T11:42:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f3fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f10fc09d1d0 0x7f10fc09d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60700/0 (socket says 172.21.15.5:60700) 2024-09-13T11:42:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f3fff640 1 -- 172.21.15.5:0/3315017940 learned_addr learned my addr 172.21.15.5:0/3315017940 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:42:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 1 --2- 172.21.15.5:0/3315017940 >> v2:172.21.15.39:3300/0 conn(0x7f10fc134810 0x7f10fc134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2384695687 0 0) 0x7f10fc09ce60 con 0x7f10fc09d1d0 2024-09-13T11:42:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient(hunting): _negotiate 2024-09-13T11:42:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient(hunting): my global_id is 4620 2024-09-13T11:42:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10e8003530 con 0x7f10fc09d1d0 2024-09-13T11:42:12.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): get_auth_request con 0x7f10fc134810 auth_method 0 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 509958436 0 0) 0x7f10e8003530 con 0x7f10fc09d1d0 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 >> v2:172.21.15.39:3300/0 conn(0x7f10fc134810 msgr2=0x7f10fc134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 --2- 172.21.15.5:0/3315017940 >> v2:172.21.15.39:3300/0 conn(0x7f10fc134810 0x7f10fc134bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 >> v1:172.21.15.39:6789/0 conn(0x7f10fc09dcf0 legacy=0x7f10fc13c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient: _finish_hunting 0 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:12.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10fc135130 con 0x7f10fc09d1d0 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient: _finish_auth 0 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient: _check_auth_tickets 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 5 monclient: authenticate success, global_id 4620 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3199362888 0 0) 0x7f10e4001700 con 0x7f10fc09d1d0 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient: dump: 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.111 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 >> v1:172.21.15.5:6789/0 conn(0x7f10fc09d1d0 legacy=0x7f10fc09d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 --2- 172.21.15.5:0/3315017940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10e8003ab0 0x7f10e8003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient(hunting): picked mon.a con 0x7f10e8003ab0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 --2- 172.21.15.5:0/3315017940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e8004810 0x7f10e8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient(hunting): picked mon.b con 0x7f10e8004810 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 --2- 172.21.15.5:0/3315017940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10e80053d0 0x7f10e800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient(hunting): picked mon.c con 0x7f10e80053d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:12.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f10e800c260 con 0x7f10e8003ab0 2024-09-13T11:42:12.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f10e800c410 con 0x7f10e8004810 2024-09-13T11:42:12.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f10e800c610 con 0x7f10e80053d0 2024-09-13T11:42:12.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:12.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f9d9b640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:42:12.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f1100bea640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:12.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 1 --2- 172.21.15.5:0/3315017940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10e8003ab0 0x7f10e8003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): get_auth_request con 0x7f10e8003ab0 auth_method 0 2024-09-13T11:42:12.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:42:12.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fad9d640 1 --2- 172.21.15.5:0/3315017940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10e80053d0 0x7f10e800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fad9d640 10 monclient(hunting): get_auth_request con 0x7f10e80053d0 auth_method 0 2024-09-13T11:42:12.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f3fff640 1 --2- 172.21.15.5:0/3315017940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e8004810 0x7f10e8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f3fff640 10 monclient(hunting): get_auth_request con 0x7f10e8004810 auth_method 0 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.099+0000 7f10f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 10 monclient(hunting): handle_auth_done global_id 4620 payload 293 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 1 -- 172.21.15.5:0/3315017940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10e80053d0 msgr2=0x7f10e800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 1 --2- 172.21.15.5:0/3315017940 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10e80053d0 0x7f10e800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 1 -- 172.21.15.5:0/3315017940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e8004810 msgr2=0x7f10e8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 1 --2- 172.21.15.5:0/3315017940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e8004810 0x7f10e8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 10 monclient: _finish_hunting 0 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 1 monclient: found mon.a 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 1 -- 172.21.15.5:0/3315017940 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e8003530 con 0x7f10e8003ab0 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 10 monclient: _finish_auth 0 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 10 monclient: _check_auth_tickets 2024-09-13T11:42:12.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:12.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fad9d640 1 -- 172.21.15.5:0/3315017940 reap_dead start 2024-09-13T11:42:12.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10f3fff640 1 --2- 172.21.15.5:0/3315017940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e8004810 0x7f10e8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:42:12.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10fa59c640 1 --2- 172.21.15.5:0/3315017940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10e8003ab0 0x7f10e8003ed0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f10ec009ae0 tx=0x7f10ec01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd07511d02a2ac2e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0096b0 con 0x7f10e8003ab0 2024-09-13T11:42:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10f9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10f9d9b640 10 monclient: dump: 2024-09-13T11:42:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.119 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ec0a6450 con 0x7f10e8003ab0 2024-09-13T11:42:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10f9d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10f9d9b640 1 -- 172.21.15.5:0/3315017940 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0a6840 con 0x7f10e8003ab0 2024-09-13T11:42:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 4 set_mon_vals no callback set 2024-09-13T11:42:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10f9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f10f9d9b640 10 monclient: dump: 2024-09-13T11:42:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.120 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.120 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.120 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.120 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.120 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.120 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.120 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.121 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 10 monclient: shutdown 2024-09-13T11:42:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 1 -- 172.21.15.5:0/3315017940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10e8003ab0 msgr2=0x7f10e8003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 1 --2- 172.21.15.5:0/3315017940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10e8003ab0 0x7f10e8003ed0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f10ec009ae0 tx=0x7f10ec01ca70 comp rx=0 tx=0).stop 2024-09-13T11:42:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 1 -- 172.21.15.5:0/3315017940 shutdown_connections 2024-09-13T11:42:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 1 --2- 172.21.15.5:0/3315017940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10e8003ab0 0x7f10e8003ed0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 1 -- 172.21.15.5:0/3315017940 >> 172.21.15.5:0/3315017940 conn(0x7f10fc0a2930 msgr2=0x7f10fc0a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 1 -- 172.21.15.5:0/3315017940 shutdown_connections 2024-09-13T11:42:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 1 -- 172.21.15.5:0/3315017940 wait complete. 2024-09-13T11:42:12.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 10 monclient: build_initial_monmap 2024-09-13T11:42:12.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 10 monclient: monmap: 2024-09-13T11:42:12.126 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:12.126 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:12.126 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:12.126 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:12.126 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:12.126 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.126 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:12.126 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.103+0000 7f1100bea640 1 Processor -- start 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 1 -- start start 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient: init 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10e8003ab0 0x7f10fc1cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient(hunting): picked mon.noname-c con 0x7f10e8003ab0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10fc127be0 0x7f10fc1339a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient(hunting): picked mon.noname-a con 0x7f10fc127be0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10fc133ee0 0x7f10fc1cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient(hunting): picked mon.noname-b con 0x7f10fc133ee0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:12.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f10fc09d940 con 0x7f10fc127be0 2024-09-13T11:42:12.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f10fc09d7c0 con 0x7f10fc133ee0 2024-09-13T11:42:12.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f10fc09dac0 con 0x7f10e8003ab0 2024-09-13T11:42:12.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:12.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient(hunting): authenticate will time out at 1135.456563s 2024-09-13T11:42:12.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f3fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10fc127be0 0x7f10fc1339a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10e8003ab0 0x7f10fc1cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10e8003ab0 0x7f10fc1cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:55032/0 (socket says 172.21.15.5:55032) 2024-09-13T11:42:12.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f3fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10fc127be0 0x7f10fc1339a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42968/0 (socket says 172.21.15.5:42968) 2024-09-13T11:42:12.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 1 -- 172.21.15.5:0/11039208 learned_addr learned my addr 172.21.15.5:0/11039208 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:12.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 10 monclient(hunting): get_auth_request con 0x7f10e8003ab0 auth_method 0 2024-09-13T11:42:12.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f3fff640 10 monclient(hunting): get_auth_request con 0x7f10fc127be0 auth_method 0 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fad9d640 1 --2- 172.21.15.5:0/11039208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10fc133ee0 0x7f10fc1cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fad9d640 10 monclient(hunting): get_auth_request con 0x7f10fc133ee0 auth_method 0 2024-09-13T11:42:12.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 10 monclient(hunting): handle_auth_done global_id 4466 payload 824 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 1 -- 172.21.15.5:0/11039208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10fc133ee0 msgr2=0x7f10fc1cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 1 --2- 172.21.15.5:0/11039208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10fc133ee0 0x7f10fc1cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 1 -- 172.21.15.5:0/11039208 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10fc127be0 msgr2=0x7f10fc1339a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 1 --2- 172.21.15.5:0/11039208 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10fc127be0 0x7f10fc1339a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 10 monclient: _finish_hunting 0 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 1 monclient: found mon.noname-c 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 1 -- 172.21.15.5:0/11039208 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10fc1293e0 con 0x7f10e8003ab0 2024-09-13T11:42:12.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10fa59c640 1 --2- 172.21.15.5:0/11039208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10e8003ab0 0x7f10fc1cc4e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01c8a0 tx=0x7f10e8005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 1 -- 172.21.15.5:0/11039208 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0c0070 con 0x7f10e8003ab0 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 10 monclient: dump: 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.132 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 10 monclient: _finish_auth 0 2024-09-13T11:42:12.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:42:12.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:12.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 1 -- 172.21.15.5:0/11039208 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ec00f000 con 0x7f10e8003ab0 2024-09-13T11:42:12.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 5 monclient: authenticate success, global_id 4466 2024-09-13T11:42:12.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient: _renew_subs 2024-09-13T11:42:12.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 1 -- 172.21.15.5:0/11039208 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10fc127080 con 0x7f10e8003ab0 2024-09-13T11:42:12.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:12.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 1 -- 172.21.15.5:0/11039208 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0a6e20 con 0x7f10e8003ab0 2024-09-13T11:42:12.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 10 monclient: dump: 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient: _renew_subs 2024-09-13T11:42:12.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 1 -- 172.21.15.5:0/11039208 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f10fc128b40 con 0x7f10e8003ab0 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f959a640 4 set_mon_vals no callback set 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 1 -- 172.21.15.5:0/11039208 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f10ec0c3070 con 0x7f10e8003ab0 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 1 --2- 172.21.15.5:0/11039208 >> v2:172.21.15.39:6812/1340838781 conn(0x7f10dc03b2a0 0x7f10dc03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f1100bea640 1 -- 172.21.15.5:0/11039208 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10fc0a13c0 con 0x7f10e8003ab0 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f2ffd640 1 -- 172.21.15.5:0/11039208 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f10ec0e8ac0 con 0x7f10e8003ab0 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f3fff640 1 --2- 172.21.15.5:0/11039208 >> v2:172.21.15.39:6812/1340838781 conn(0x7f10dc03b2a0 0x7f10dc03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f3fff640 10 monclient: get_auth_request con 0x7f10dc03b2a0 auth_method 0 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.107+0000 7f10f3fff640 1 --2- 172.21.15.5:0/11039208 >> v2:172.21.15.39:6812/1340838781 conn(0x7f10dc03b2a0 0x7f10dc03d760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f10fc1cd5a0 tx=0x7f10e4007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.111+0000 7f10f2ffd640 1 -- 172.21.15.5:0/11039208 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f10ec0ac020 con 0x7f10e8003ab0 2024-09-13T11:42:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.111+0000 7f10f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:12.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.111+0000 7f10f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:12.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.283+0000 7f1100bea640 1 -- 172.21.15.5:0/11039208 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f10fc06dbf0 con 0x7f10dc03b2a0 2024-09-13T11:42:12.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.283+0000 7f10f2ffd640 1 -- 172.21.15.5:0/11039208 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f10fc06dbf0 con 0x7f10dc03b2a0 2024-09-13T11:42:12.288 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T11:42:12.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 1 -- 172.21.15.5:0/11039208 >> v2:172.21.15.39:6812/1340838781 conn(0x7f10dc03b2a0 msgr2=0x7f10dc03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 1 --2- 172.21.15.5:0/11039208 >> v2:172.21.15.39:6812/1340838781 conn(0x7f10dc03b2a0 0x7f10dc03d760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f10fc1cd5a0 tx=0x7f10e4007040 comp rx=0 tx=0).stop 2024-09-13T11:42:12.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 10 monclient: shutdown 2024-09-13T11:42:12.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:12.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 1 -- 172.21.15.5:0/11039208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10e8003ab0 msgr2=0x7f10fc1cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 1 --2- 172.21.15.5:0/11039208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10e8003ab0 0x7f10fc1cc4e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01c8a0 tx=0x7f10e8005840 comp rx=0 tx=0).stop 2024-09-13T11:42:12.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 1 -- 172.21.15.5:0/11039208 shutdown_connections 2024-09-13T11:42:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 1 --2- 172.21.15.5:0/11039208 >> v2:172.21.15.39:6812/1340838781 conn(0x7f10dc03b2a0 0x7f10dc03d760 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 1 --2- 172.21.15.5:0/11039208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10fc133ee0 0x7f10fc1cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 1 --2- 172.21.15.5:0/11039208 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10fc127be0 0x7f10fc1339a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 1 --2- 172.21.15.5:0/11039208 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10e8003ab0 0x7f10fc1cc4e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 1 -- 172.21.15.5:0/11039208 >> 172.21.15.5:0/11039208 conn(0x7f10fc0a2930 msgr2=0x7f10fc072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 1 -- 172.21.15.5:0/11039208 shutdown_connections 2024-09-13T11:42:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.287+0000 7f1100bea640 1 -- 172.21.15.5:0/11039208 wait complete. 2024-09-13T11:42:12.306 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T11:42:12.307 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:42:12.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient: get_monmap_and_config 2024-09-13T11:42:12.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient: build_initial_monmap 2024-09-13T11:42:12.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient: monmap: 2024-09-13T11:42:12.430 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:12.430 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.430 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:12.429418+0000 2024-09-13T11:42:12.430 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:12.429418+0000 2024-09-13T11:42:12.430 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:12.430 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.430 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:12.430 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:12.430 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:12.430 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:12.430 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:12.430 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 1 Processor -- start 2024-09-13T11:42:12.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 1 -- start start 2024-09-13T11:42:12.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient: init 2024-09-13T11:42:12.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:12.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-13T11:42:12.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 0x7fb3840a3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient(hunting): picked mon.noname-b con 0x7fb3840a3910 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:12.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb38409ffc0 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:12.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb3840a4230 0x7fb3841381c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient(hunting): picked mon.noname-c con 0x7fb3840a4230 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:12.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb38406f2c0 con 0x7fb38409ffc0 2024-09-13T11:42:12.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fb38406ee80 con 0x7fb3840a4230 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fb38409eee0 con 0x7fb3840a3910 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3896c6640 10 monclient(hunting): authenticate will time out at 1135.778906s 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 0x7fb3840a3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 0x7fb3840a3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:55036/0 (socket says 172.21.15.5:55036) 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 1 -- 172.21.15.5:0/514703511 learned_addr learned my addr 172.21.15.5:0/514703511 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 10 monclient(hunting): get_auth_request con 0x7fb3840a3910 auth_method 0 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3837fe640 1 --2- 172.21.15.5:0/514703511 >> v2:172.21.15.39:3300/0 conn(0x7fb3840a4230 0x7fb3841381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3837fe640 10 monclient(hunting): get_auth_request con 0x7fb3840a4230 auth_method 0 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3837fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb3837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 10 monclient(hunting): handle_auth_done global_id 4469 payload 274 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 1 -- 172.21.15.5:0/514703511 >> v2:172.21.15.39:3300/0 conn(0x7fb3840a4230 msgr2=0x7fb3841381c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 1 --2- 172.21.15.5:0/514703511 >> v2:172.21.15.39:3300/0 conn(0x7fb3840a4230 0x7fb3841381c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 1 -- 172.21.15.5:0/514703511 >> v1:172.21.15.39:6789/0 conn(0x7fb38409ffc0 legacy=0x7fb384137ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 10 monclient: _finish_hunting 0 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 1 monclient: found mon.noname-b 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:12.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.427+0000 7fb382ffd640 1 -- 172.21.15.5:0/514703511 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3840705e0 con 0x7fb3840a3910 2024-09-13T11:42:12.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb382ffd640 1 --2- 172.21.15.5:0/514703511 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 0x7fb3840a3cf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb374009870 tx=0x7fb37401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5d3c2ea6bf71efc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 1 -- 172.21.15.5:0/514703511 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3740b1070 con 0x7fb3840a3910 2024-09-13T11:42:12.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:42:12.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 10 monclient: dump: 2024-09-13T11:42:12.441 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.441 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 10 monclient: _finish_auth 0 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 10 monclient: _check_auth_tickets 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 1 -- 172.21.15.5:0/514703511 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3740ac620 con 0x7fb3840a3910 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 5 monclient: authenticate success, global_id 4469 2024-09-13T11:42:12.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 1 -- 172.21.15.5:0/514703511 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3740ac9f0 con 0x7fb3840a3910 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 4 set_mon_vals no callback set 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3827fc640 10 monclient: dump: 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.443 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 10 monclient: shutdown 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 1 -- 172.21.15.5:0/514703511 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 msgr2=0x7fb3840a3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 1 --2- 172.21.15.5:0/514703511 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 0x7fb3840a3cf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb374009870 tx=0x7fb37401c420 comp rx=0 tx=0).stop 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 1 -- 172.21.15.5:0/514703511 shutdown_connections 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 1 --2- 172.21.15.5:0/514703511 >> v2:172.21.15.39:3300/0 conn(0x7fb3840a4230 0x7fb3841381c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 1 --2- 172.21.15.5:0/514703511 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 0x7fb3840a3cf0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 1 -- 172.21.15.5:0/514703511 >> 172.21.15.5:0/514703511 conn(0x7fb3840a8590 msgr2=0x7fb3840a4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 1 -- 172.21.15.5:0/514703511 shutdown_connections 2024-09-13T11:42:12.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 1 -- 172.21.15.5:0/514703511 wait complete. 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 10 monclient: build_initial_monmap 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 10 monclient: monmap: 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 1 Processor -- start 2024-09-13T11:42:12.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 1 -- start start 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.431+0000 7fb3896c6640 10 monclient: init 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb38409ffc0 0x7fb3841cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient(hunting): picked mon.noname-b con 0x7fb38409ffc0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 0x7fb3841cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient(hunting): picked mon.noname-c con 0x7fb3840a3910 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb3840a4230 0x7fb3841d0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient(hunting): picked mon.noname-a con 0x7fb3840a4230 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb384138880 con 0x7fb3840a4230 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb384138700 con 0x7fb38409ffc0 2024-09-13T11:42:12.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fb384142530 con 0x7fb3840a3910 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient(hunting): authenticate will time out at 1135.784387s 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 0x7fb3841cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3837fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb3840a4230 0x7fb3841d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 0x7fb3841cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:55042/0 (socket says 172.21.15.5:55042) 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 1 -- 172.21.15.5:0/1616752789 learned_addr learned my addr 172.21.15.5:0/1616752789 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 10 monclient(hunting): get_auth_request con 0x7fb3840a3910 auth_method 0 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3837fe640 10 monclient(hunting): get_auth_request con 0x7fb3840a4230 auth_method 0 2024-09-13T11:42:12.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3837fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb382ffd640 1 --2- 172.21.15.5:0/1616752789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb38409ffc0 0x7fb3841cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb382ffd640 10 monclient(hunting): get_auth_request con 0x7fb38409ffc0 auth_method 0 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb382ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb382ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb382ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 10 monclient(hunting): handle_auth_done global_id 4472 payload 824 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 1 -- 172.21.15.5:0/1616752789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb38409ffc0 msgr2=0x7fb3841cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 1 --2- 172.21.15.5:0/1616752789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb38409ffc0 0x7fb3841cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 1 -- 172.21.15.5:0/1616752789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb3840a4230 msgr2=0x7fb3841d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 1 --2- 172.21.15.5:0/1616752789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb3840a4230 0x7fb3841d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 10 monclient: _finish_hunting 0 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 1 monclient: found mon.noname-c 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 1 -- 172.21.15.5:0/1616752789 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb38409f0f0 con 0x7fb3840a3910 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb382ffd640 1 --2- 172.21.15.5:0/1616752789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb38409ffc0 0x7fb3841cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3837fe640 1 --2- 172.21.15.5:0/1616752789 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb3840a4230 0x7fb3841d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37bfff640 1 --2- 172.21.15.5:0/1616752789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 0x7fb3841cfcf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00c440 tx=0x7fb36c00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 1 -- 172.21.15.5:0/1616752789 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c016020 con 0x7fb3840a3910 2024-09-13T11:42:12.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 10 monclient: dump: 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 10 monclient: _finish_auth 0 2024-09-13T11:42:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 1 -- 172.21.15.5:0/1616752789 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb36c012830 con 0x7fb3840a3910 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 1 -- 172.21.15.5:0/1616752789 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c012c20 con 0x7fb3840a3910 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 10 monclient: dump: 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.452 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.452 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.452 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 5 monclient: authenticate success, global_id 4472 2024-09-13T11:42:12.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient: _renew_subs 2024-09-13T11:42:12.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3817fa640 4 set_mon_vals no callback set 2024-09-13T11:42:12.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 1 -- 172.21.15.5:0/1616752789 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3840a65e0 con 0x7fb3840a3910 2024-09-13T11:42:12.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient: _renew_subs 2024-09-13T11:42:12.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 1 -- 172.21.15.5:0/1616752789 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb38413f250 con 0x7fb3840a3910 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3896c6640 1 -- 172.21.15.5:0/1616752789 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb3840a3cf0 con 0x7fb3840a3910 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 1 -- 172.21.15.5:0/1616752789 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fb36c019020 con 0x7fb3840a3910 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 1 --2- 172.21.15.5:0/1616752789 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb36403b2d0 0x7fb36403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 1 -- 172.21.15.5:0/1616752789 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7fb36c04e930 con 0x7fb3840a3910 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 1 --2- 172.21.15.5:0/1616752789 >> v2:172.21.15.5:6808/1447734001 conn(0x7fb364040de0 0x7fb364043200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 1 -- 172.21.15.5:0/1616752789 --> v2:172.21.15.5:6808/1447734001 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb36c01eeb0 con 0x7fb364040de0 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 1 -- 172.21.15.5:0/1616752789 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb36c0532c0 con 0x7fb3840a3910 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3837fe640 1 --2- 172.21.15.5:0/1616752789 >> v2:172.21.15.5:6808/1447734001 conn(0x7fb364040de0 0x7fb364043200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3837fe640 10 monclient: get_auth_request con 0x7fb364040de0 auth_method 0 2024-09-13T11:42:12.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb382ffd640 1 --2- 172.21.15.5:0/1616752789 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb36403b2d0 0x7fb36403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb382ffd640 10 monclient: get_auth_request con 0x7fb36403b2d0 auth_method 0 2024-09-13T11:42:12.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb3837fe640 1 --2- 172.21.15.5:0/1616752789 >> v2:172.21.15.5:6808/1447734001 conn(0x7fb364040de0 0x7fb364043200 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb382ffd640 1 --2- 172.21.15.5:0/1616752789 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb36403b2d0 0x7fb36403d790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb3740ae000 tx=0x7fb3740a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.435+0000 7fb37b7fe640 1 -- 172.21.15.5:0/1616752789 <== osd.1 v2:172.21.15.5:6808/1447734001 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb36c01eeb0 con 0x7fb364040de0 2024-09-13T11:42:12.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 1 -- 172.21.15.5:0/1616752789 --> v2:172.21.15.5:6808/1447734001 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb38413df90 con 0x7fb364040de0 2024-09-13T11:42:12.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb37b7fe640 1 -- 172.21.15.5:0/1616752789 <== osd.1 v2:172.21.15.5:6808/1447734001 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb38413df90 con 0x7fb364040de0 2024-09-13T11:42:12.455 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:42:12.455 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:42:12.455 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:42:12.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 1 -- 172.21.15.5:0/1616752789 >> v2:172.21.15.5:6808/1447734001 conn(0x7fb364040de0 msgr2=0x7fb364043200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 1 --2- 172.21.15.5:0/1616752789 >> v2:172.21.15.5:6808/1447734001 conn(0x7fb364040de0 0x7fb364043200 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 1 -- 172.21.15.5:0/1616752789 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb36403b2d0 msgr2=0x7fb36403d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 1 --2- 172.21.15.5:0/1616752789 >> v2:172.21.15.39:6812/1340838781 conn(0x7fb36403b2d0 0x7fb36403d790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb3740ae000 tx=0x7fb3740a7000 comp rx=0 tx=0).stop 2024-09-13T11:42:12.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 10 monclient: shutdown 2024-09-13T11:42:12.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:12.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 1 -- 172.21.15.5:0/1616752789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 msgr2=0x7fb3841cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 1 --2- 172.21.15.5:0/1616752789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb3840a3910 0x7fb3841cfcf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00c440 tx=0x7fb36c00c900 comp rx=0 tx=0).stop 2024-09-13T11:42:12.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3837fe640 1 -- 172.21.15.5:0/1616752789 reap_dead start 2024-09-13T11:42:12.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 1 -- 172.21.15.5:0/1616752789 shutdown_connections 2024-09-13T11:42:12.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 1 -- 172.21.15.5:0/1616752789 >> 172.21.15.5:0/1616752789 conn(0x7fb3840a8590 msgr2=0x7fb3840a6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:12.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 1 -- 172.21.15.5:0/1616752789 shutdown_connections 2024-09-13T11:42:12.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.451+0000 7fb3896c6640 1 -- 172.21.15.5:0/1616752789 wait complete. 2024-09-13T11:42:12.469 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T11:42:12.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 10 monclient: get_monmap_and_config 2024-09-13T11:42:12.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 10 monclient: build_initial_monmap 2024-09-13T11:42:12.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 10 monclient: monmap: 2024-09-13T11:42:12.581 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:12.581 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.581 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:12.581331+0000 2024-09-13T11:42:12.582 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:12.581331+0000 2024-09-13T11:42:12.582 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:12.582 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.582 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:12.582 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:12.582 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:12.582 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:12.582 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:12.582 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 1 Processor -- start 2024-09-13T11:42:12.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 1 -- start start 2024-09-13T11:42:12.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 10 monclient: init 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03240a0ea0 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f03240a1ae0 0x7f032413a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 10 monclient(hunting): picked mon.noname-a con 0x7f03240a1ae0 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f032413ad30 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f032406f010 con 0x7f03240a0ea0 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f032409fd80 con 0x7f032413ad30 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f032406f640 con 0x7f03240a1ae0 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f032bac5640 10 monclient(hunting): authenticate will time out at 1135.930012s 2024-09-13T11:42:12.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329861640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f03240a0ea0 0x7f03240a1280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60716/0 (socket says 172.21.15.5:60716) 2024-09-13T11:42:12.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329861640 1 -- 172.21.15.5:0/2345290857 learned_addr learned my addr 172.21.15.5:0/2345290857 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:42:12.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 1 --2- 172.21.15.5:0/2345290857 >> v2:172.21.15.5:3300/0 conn(0x7f03240a1ae0 0x7f032413a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 10 monclient(hunting): get_auth_request con 0x7f03240a1ae0 auth_method 0 2024-09-13T11:42:12.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:12.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 10 monclient(hunting): handle_auth_done global_id 4629 payload 274 2024-09-13T11:42:12.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 1 -- 172.21.15.5:0/2345290857 >> v1:172.21.15.39:6789/0 conn(0x7f032413ad30 legacy=0x7f032413d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:12.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 1 -- 172.21.15.5:0/2345290857 >> v1:172.21.15.5:6789/0 conn(0x7f03240a0ea0 legacy=0x7f03240a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 10 monclient: _finish_hunting 0 2024-09-13T11:42:12.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 1 monclient: found mon.noname-a 2024-09-13T11:42:12.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:12.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:12.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.579+0000 7f0329060640 1 -- 172.21.15.5:0/2345290857 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03240a23f0 con 0x7f03240a1ae0 2024-09-13T11:42:12.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 1 --2- 172.21.15.5:0/2345290857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03240a1ae0 0x7f032413a7f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f03200097c0 tx=0x7f032001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d222adb6802a0e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 1 -- 172.21.15.5:0/2345290857 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03200b0070 con 0x7f03240a1ae0 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 10 monclient: dump: 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 10 monclient: _finish_auth 0 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 10 monclient: _check_auth_tickets 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 5 monclient: authenticate success, global_id 4629 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 1 -- 172.21.15.5:0/2345290857 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03200ab600 con 0x7f03240a1ae0 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 4 set_mon_vals no callback set 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 1 -- 172.21.15.5:0/2345290857 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03200ab9d0 con 0x7f03240a1ae0 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032885f640 10 monclient: dump: 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient: shutdown 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 -- 172.21.15.5:0/2345290857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03240a1ae0 msgr2=0x7f032413a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 --2- 172.21.15.5:0/2345290857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03240a1ae0 0x7f032413a7f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f03200097c0 tx=0x7f032001c3f0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 -- 172.21.15.5:0/2345290857 shutdown_connections 2024-09-13T11:42:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 --2- 172.21.15.5:0/2345290857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03240a1ae0 0x7f032413a7f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 -- 172.21.15.5:0/2345290857 >> 172.21.15.5:0/2345290857 conn(0x7f03240a2930 msgr2=0x7f03240a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 -- 172.21.15.5:0/2345290857 shutdown_connections 2024-09-13T11:42:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 -- 172.21.15.5:0/2345290857 wait complete. 2024-09-13T11:42:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient: build_initial_monmap 2024-09-13T11:42:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient: monmap: 2024-09-13T11:42:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 Processor -- start 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 -- start start 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient: init 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03240a0ea0 0x7f032406ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient(hunting): picked mon.noname-c con 0x7f03240a0ea0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03240a1ae0 0x7f03240720a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient(hunting): picked mon.noname-a con 0x7f03240a1ae0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f032413ad30 0x7f03241305a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient(hunting): picked mon.noname-b con 0x7f032413ad30 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0324141800 con 0x7f03240a1ae0 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0324141680 con 0x7f032413ad30 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f0324141980 con 0x7f03240a0ea0 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient(hunting): authenticate will time out at 1135.934705s 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03240a1ae0 0x7f03240720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03240a1ae0 0x7f03240720a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42998/0 (socket says 172.21.15.5:42998) 2024-09-13T11:42:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 1 -- 172.21.15.5:0/811178959 learned_addr learned my addr 172.21.15.5:0/811178959 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 10 monclient(hunting): get_auth_request con 0x7f03240a1ae0 auth_method 0 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329861640 1 --2- 172.21.15.5:0/811178959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03240a0ea0 0x7f032406ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329861640 10 monclient(hunting): get_auth_request con 0x7f03240a0ea0 auth_method 0 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329861640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329861640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329861640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329861640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329861640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329861640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032a062640 1 --2- 172.21.15.5:0/811178959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f032413ad30 0x7f03241305a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 10 monclient(hunting): handle_auth_done global_id 4635 payload 824 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 1 -- 172.21.15.5:0/811178959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03240a0ea0 msgr2=0x7f032406ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 1 --2- 172.21.15.5:0/811178959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03240a0ea0 0x7f032406ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 1 -- 172.21.15.5:0/811178959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f032413ad30 msgr2=0x7f03241305a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 1 --2- 172.21.15.5:0/811178959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f032413ad30 0x7f03241305a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 10 monclient: _finish_hunting 0 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 1 monclient: found mon.noname-a 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 1 -- 172.21.15.5:0/811178959 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032406dbf0 con 0x7f03240a1ae0 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032a062640 10 monclient: get_auth_request con 0x7f032413ad30 auth_method 0 2024-09-13T11:42:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032a062640 1 --2- 172.21.15.5:0/811178959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f032413ad30 0x7f03241305a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f0329060640 1 --2- 172.21.15.5:0/811178959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03240a1ae0 0x7f03240720a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03200ae040 tx=0x7f0320019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 1 -- 172.21.15.5:0/811178959 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03200b8030 con 0x7f03240a1ae0 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 10 monclient: dump: 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 10 monclient: _finish_auth 0 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 1 -- 172.21.15.5:0/811178959 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03200b4410 con 0x7f03240a1ae0 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 1 -- 172.21.15.5:0/811178959 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03200b0040 con 0x7f03240a1ae0 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031a7fc640 10 monclient: dump: 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.602 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 5 monclient: authenticate success, global_id 4635 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient: _renew_subs 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f031bfff640 4 set_mon_vals no callback set 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 -- 172.21.15.5:0/811178959 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0324071540 con 0x7f03240a1ae0 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient: _renew_subs 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.583+0000 7f032bac5640 1 -- 172.21.15.5:0/811178959 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0324073000 con 0x7f03240a1ae0 2024-09-13T11:42:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.587+0000 7f031a7fc640 1 -- 172.21.15.5:0/811178959 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f03200bb070 con 0x7f03240a1ae0 2024-09-13T11:42:12.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.587+0000 7f032bac5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:12.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.587+0000 7f032bac5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:12.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.587+0000 7f032bac5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:12.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.587+0000 7f032bac5640 1 -- 172.21.15.5:0/811178959 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03240a1280 con 0x7f03240a1ae0 2024-09-13T11:42:12.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.587+0000 7f031a7fc640 1 --2- 172.21.15.5:0/811178959 >> v2:172.21.15.39:6812/1340838781 conn(0x7f030803b2a0 0x7f030803d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.587+0000 7f031a7fc640 1 -- 172.21.15.5:0/811178959 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f03200e8630 con 0x7f03240a1ae0 2024-09-13T11:42:12.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.587+0000 7f0329861640 1 --2- 172.21.15.5:0/811178959 >> v2:172.21.15.39:6812/1340838781 conn(0x7f030803b2a0 0x7f030803d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.587+0000 7f0329861640 10 monclient: get_auth_request con 0x7f030803b2a0 auth_method 0 2024-09-13T11:42:12.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.587+0000 7f0329861640 1 --2- 172.21.15.5:0/811178959 >> v2:172.21.15.39:6812/1340838781 conn(0x7f030803b2a0 0x7f030803d760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0314000f70 tx=0x7f0314009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.591+0000 7f031a7fc640 1 -- 172.21.15.5:0/811178959 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f03200c2070 con 0x7f03240a1ae0 2024-09-13T11:42:12.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.591+0000 7f031a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:12.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.591+0000 7f031a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 -- 172.21.15.5:0/811178959 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f032409c760 con 0x7f030803b2a0 2024-09-13T11:42:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f031a7fc640 1 -- 172.21.15.5:0/811178959 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f032409c760 con 0x7f030803b2a0 2024-09-13T11:42:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T11:42:12.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 -- 172.21.15.5:0/811178959 >> v2:172.21.15.39:6812/1340838781 conn(0x7f030803b2a0 msgr2=0x7f030803d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 --2- 172.21.15.5:0/811178959 >> v2:172.21.15.39:6812/1340838781 conn(0x7f030803b2a0 0x7f030803d760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0314000f70 tx=0x7f0314009040 comp rx=0 tx=0).stop 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 10 monclient: shutdown 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 -- 172.21.15.5:0/811178959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03240a1ae0 msgr2=0x7f03240720a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 --2- 172.21.15.5:0/811178959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03240a1ae0 0x7f03240720a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03200ae040 tx=0x7f0320019190 comp rx=0 tx=0).stop 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 -- 172.21.15.5:0/811178959 shutdown_connections 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 --2- 172.21.15.5:0/811178959 >> v2:172.21.15.39:6812/1340838781 conn(0x7f030803b2a0 0x7f030803d760 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 --2- 172.21.15.5:0/811178959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f032413ad30 0x7f03241305a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 --2- 172.21.15.5:0/811178959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03240a1ae0 0x7f03240720a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 --2- 172.21.15.5:0/811178959 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03240a0ea0 0x7f032406ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 -- 172.21.15.5:0/811178959 >> 172.21.15.5:0/811178959 conn(0x7f03240a2930 msgr2=0x7f03240a49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 -- 172.21.15.5:0/811178959 shutdown_connections 2024-09-13T11:42:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.755+0000 7f032bac5640 1 -- 172.21.15.5:0/811178959 wait complete. 2024-09-13T11:42:12.776 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T11:42:12.776 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:42:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 10 monclient: get_monmap_and_config 2024-09-13T11:42:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 10 monclient: build_initial_monmap 2024-09-13T11:42:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 10 monclient: monmap: 2024-09-13T11:42:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:12.902159+0000 2024-09-13T11:42:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:12.902159+0000 2024-09-13T11:42:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:12.903 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 1 Processor -- start 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 1 -- start start 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 10 monclient: init 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f506009d3e0 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f506009ded0 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f506013b1b0 0x7f506013d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 10 monclient(hunting): picked mon.noname-c con 0x7f506013b1b0 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f506009fb70 con 0x7f506009d3e0 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f506009e8a0 con 0x7f506009ded0 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f5060074540 con 0x7f506013b1b0 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f50651ec640 10 monclient(hunting): authenticate will time out at 1136.250919s 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505ed9d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f506009d3e0 0x7f506009d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60718/0 (socket says 172.21.15.5:60718) 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505ed9d640 1 -- 172.21.15.5:0/2593287724 learned_addr learned my addr 172.21.15.5:0/2593287724 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505f59e640 1 --2- 172.21.15.5:0/2593287724 >> v2:172.21.15.39:3300/0 conn(0x7f506013b1b0 0x7f506013d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2347369960 0 0) 0x7f5050001120 con 0x7f506009d3e0 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient(hunting): _negotiate 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient(hunting): my global_id is 4638 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f504c003530 con 0x7f506009d3e0 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505f59e640 10 monclient(hunting): get_auth_request con 0x7f506013b1b0 auth_method 0 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505f59e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2115376643 0 0) 0x7f504c003530 con 0x7f506009d3e0 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 >> v2:172.21.15.39:3300/0 conn(0x7f506013b1b0 msgr2=0x7f506013d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 --2- 172.21.15.5:0/2593287724 >> v2:172.21.15.39:3300/0 conn(0x7f506013b1b0 0x7f506013d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 >> v1:172.21.15.39:6789/0 conn(0x7f506009ded0 legacy=0x7f5060139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient: _finish_hunting 0 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 monclient: found mon.noname-a-legacy 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50600a1ca0 con 0x7f506009d3e0 2024-09-13T11:42:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient: _finish_auth 0 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient: _check_auth_tickets 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3199362888 0 0) 0x7f50500018d0 con 0x7f506009d3e0 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient: dump: 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 >> v1:172.21.15.5:6789/0 conn(0x7f506009d3e0 legacy=0x7f506009d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 --2- 172.21.15.5:0/2593287724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f504c003a60 0x7f504c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient(hunting): picked mon.c con 0x7f504c003a60 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 --2- 172.21.15.5:0/2593287724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f504c004810 0x7f504c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient(hunting): picked mon.b con 0x7f504c004810 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 --2- 172.21.15.5:0/2593287724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f504c0053d0 0x7f504c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient(hunting): picked mon.a con 0x7f504c0053d0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f504c00c210 con 0x7f504c0053d0 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f504c00c3c0 con 0x7f504c004810 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f504c00c5c0 con 0x7f504c003a60 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.899+0000 7f505e59c640 10 monclient(hunting): _finish_auth 0 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 1 --2- 172.21.15.5:0/2593287724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f504c003a60 0x7f504c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505f59e640 1 --2- 172.21.15.5:0/2593287724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f504c0053d0 0x7f504c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 10 monclient(hunting): get_auth_request con 0x7f504c003a60 auth_method 0 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:42:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505f59e640 10 monclient(hunting): get_auth_request con 0x7f504c0053d0 auth_method 0 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505f59e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f5057fff640 1 --2- 172.21.15.5:0/2593287724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f504c004810 0x7f504c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505f59e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f5057fff640 10 monclient(hunting): get_auth_request con 0x7f504c004810 auth_method 0 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f5057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f5057fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f5057fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 10 monclient(hunting): handle_auth_done global_id 4638 payload 293 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 1 -- 172.21.15.5:0/2593287724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f504c004810 msgr2=0x7f504c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 1 --2- 172.21.15.5:0/2593287724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f504c004810 0x7f504c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 1 -- 172.21.15.5:0/2593287724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f504c0053d0 msgr2=0x7f504c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 1 --2- 172.21.15.5:0/2593287724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f504c0053d0 0x7f504c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 10 monclient: _finish_hunting 0 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 1 monclient: found mon.c 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 1 -- 172.21.15.5:0/2593287724 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f504c003530 con 0x7f504c003a60 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 10 monclient: _finish_auth 0 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 10 monclient: _check_auth_tickets 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 5 monclient: authenticate success, global_id 4638 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505f59e640 1 --2- 172.21.15.5:0/2593287724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f504c0053d0 0x7f504c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505f59e640 1 -- 172.21.15.5:0/2593287724 reap_dead start 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505ed9d640 1 --2- 172.21.15.5:0/2593287724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f504c003a60 0x7f504c003e80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f505000c490 tx=0x7f505001cb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0a2340a4f87d108 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505000c0e0 con 0x7f504c003a60 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505e59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505e59c640 10 monclient: dump: 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50500a9020 con 0x7f504c003a60 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505e59c640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505e59c640 1 -- 172.21.15.5:0/2593287724 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050005b20 con 0x7f504c003a60 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 4 set_mon_vals no callback set 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505e59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f505e59c640 10 monclient: dump: 2024-09-13T11:42:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 10 monclient: shutdown 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 1 -- 172.21.15.5:0/2593287724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f504c003a60 msgr2=0x7f504c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 1 --2- 172.21.15.5:0/2593287724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f504c003a60 0x7f504c003e80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f505000c490 tx=0x7f505001cb60 comp rx=0 tx=0).stop 2024-09-13T11:42:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 1 -- 172.21.15.5:0/2593287724 shutdown_connections 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 1 --2- 172.21.15.5:0/2593287724 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f504c003a60 0x7f504c003e80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 1 -- 172.21.15.5:0/2593287724 >> 172.21.15.5:0/2593287724 conn(0x7f50600a6780 msgr2=0x7f50600a2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 1 -- 172.21.15.5:0/2593287724 shutdown_connections 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 1 -- 172.21.15.5:0/2593287724 wait complete. 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 10 monclient: build_initial_monmap 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.903+0000 7f50651ec640 10 monclient: monmap: 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:12.921 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:12.921 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 1 Processor -- start 2024-09-13T11:42:12.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 1 -- start start 2024-09-13T11:42:12.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient: init 2024-09-13T11:42:12.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:12.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:42:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f504c003a60 0x7f50601cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient(hunting): picked mon.noname-c con 0x7f504c003a60 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5060133680 0x7f5060133b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient(hunting): picked mon.noname-a con 0x7f5060133680 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506011d2c0 0x7f506011d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient(hunting): picked mon.noname-b con 0x7f506011d2c0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f5060134f60 con 0x7f5060133680 2024-09-13T11:42:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5060134de0 con 0x7f506011d2c0 2024-09-13T11:42:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f50601350e0 con 0x7f504c003a60 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient(hunting): authenticate will time out at 1136.256888s 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f504c003a60 0x7f50601cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f504c003a60 0x7f50601cfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:55070/0 (socket says 172.21.15.5:55070) 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f5057fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5060133680 0x7f5060133b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 1 -- 172.21.15.5:0/2864246138 learned_addr learned my addr 172.21.15.5:0/2864246138 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 10 monclient(hunting): get_auth_request con 0x7f504c003a60 auth_method 0 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f5057fff640 10 monclient(hunting): get_auth_request con 0x7f5060133680 auth_method 0 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f5057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f5057fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f5057fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f5057fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f5057fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f5057fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505f59e640 1 --2- 172.21.15.5:0/2864246138 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506011d2c0 0x7f506011d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505f59e640 10 monclient(hunting): get_auth_request con 0x7f506011d2c0 auth_method 0 2024-09-13T11:42:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505f59e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 10 monclient(hunting): handle_auth_done global_id 4478 payload 824 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 1 -- 172.21.15.5:0/2864246138 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506011d2c0 msgr2=0x7f506011d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 1 --2- 172.21.15.5:0/2864246138 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f506011d2c0 0x7f506011d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 1 -- 172.21.15.5:0/2864246138 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5060133680 msgr2=0x7f5060133b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 1 --2- 172.21.15.5:0/2864246138 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5060133680 0x7f5060133b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 10 monclient: _finish_hunting 0 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 1 monclient: found mon.noname-c 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 1 -- 172.21.15.5:0/2864246138 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5060134040 con 0x7f504c003a60 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f5057fff640 1 --2- 172.21.15.5:0/2864246138 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5060133680 0x7f5060133b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505ed9d640 1 --2- 172.21.15.5:0/2864246138 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f504c003a60 0x7f50601cfa50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5050000c10 tx=0x7f504c005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 1 -- 172.21.15.5:0/2864246138 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50500047f0 con 0x7f504c003a60 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 10 monclient: dump: 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 10 monclient: _finish_auth 0 2024-09-13T11:42:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 10 monclient: _check_auth_tickets 2024-09-13T11:42:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 1 -- 172.21.15.5:0/2864246138 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5050004990 con 0x7f504c003a60 2024-09-13T11:42:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 5 monclient: authenticate success, global_id 4478 2024-09-13T11:42:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient: _renew_subs 2024-09-13T11:42:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 1 -- 172.21.15.5:0/2864246138 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f506011ece0 con 0x7f504c003a60 2024-09-13T11:42:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 1 -- 172.21.15.5:0/2864246138 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050004d40 con 0x7f504c003a60 2024-09-13T11:42:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 10 monclient: dump: 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505dd9b640 4 set_mon_vals no callback set 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient: _renew_subs 2024-09-13T11:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 1 -- 172.21.15.5:0/2864246138 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f506011dc90 con 0x7f504c003a60 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50651ec640 1 -- 172.21.15.5:0/2864246138 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f506009d0d0 con 0x7f504c003a60 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 1 -- 172.21.15.5:0/2864246138 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f50500a56a0 con 0x7f504c003a60 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 1 --2- 172.21.15.5:0/2864246138 >> v2:172.21.15.39:6812/1340838781 conn(0x7f504003b2a0 0x7f504003d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 1 -- 172.21.15.5:0/2864246138 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f50500e7d10 con 0x7f504c003a60 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 1 --2- 172.21.15.5:0/2864246138 >> v2:172.21.15.5:6800/1907498696 conn(0x7f5040040db0 0x7f50400431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 1 -- 172.21.15.5:0/2864246138 --> v2:172.21.15.5:6800/1907498696 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50500a9e90 con 0x7f5040040db0 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 1 -- 172.21.15.5:0/2864246138 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f50500ea3e0 con 0x7f504c003a60 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f50577fe640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505f59e640 1 --2- 172.21.15.5:0/2864246138 >> v2:172.21.15.5:6800/1907498696 conn(0x7f5040040db0 0x7f50400431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505f59e640 10 monclient: get_auth_request con 0x7f5040040db0 auth_method 0 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f5057fff640 1 --2- 172.21.15.5:0/2864246138 >> v2:172.21.15.39:6812/1340838781 conn(0x7f504003b2a0 0x7f504003d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f5057fff640 10 monclient: get_auth_request con 0x7f504003b2a0 auth_method 0 2024-09-13T11:42:12.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.907+0000 7f505f59e640 1 --2- 172.21.15.5:0/2864246138 >> v2:172.21.15.5:6800/1907498696 conn(0x7f5040040db0 0x7f50400431d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.911+0000 7f5057fff640 1 --2- 172.21.15.5:0/2864246138 >> v2:172.21.15.39:6812/1340838781 conn(0x7f504003b2a0 0x7f504003d760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5060073c20 tx=0x7f5048009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:12.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.911+0000 7f50577fe640 1 -- 172.21.15.5:0/2864246138 <== osd.2 v2:172.21.15.5:6800/1907498696 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50500a9e90 con 0x7f5040040db0 2024-09-13T11:42:12.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 1 -- 172.21.15.5:0/2864246138 --> v2:172.21.15.5:6800/1907498696 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f50600a0800 con 0x7f5040040db0 2024-09-13T11:42:12.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50577fe640 1 -- 172.21.15.5:0/2864246138 <== osd.2 v2:172.21.15.5:6800/1907498696 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f50600a0800 con 0x7f5040040db0 2024-09-13T11:42:12.935 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:42:12.935 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:42:12.935 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:42:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 1 -- 172.21.15.5:0/2864246138 >> v2:172.21.15.5:6800/1907498696 conn(0x7f5040040db0 msgr2=0x7f50400431d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 1 --2- 172.21.15.5:0/2864246138 >> v2:172.21.15.5:6800/1907498696 conn(0x7f5040040db0 0x7f50400431d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 1 -- 172.21.15.5:0/2864246138 >> v2:172.21.15.39:6812/1340838781 conn(0x7f504003b2a0 msgr2=0x7f504003d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 1 --2- 172.21.15.5:0/2864246138 >> v2:172.21.15.39:6812/1340838781 conn(0x7f504003b2a0 0x7f504003d760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5060073c20 tx=0x7f5048009040 comp rx=0 tx=0).stop 2024-09-13T11:42:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 10 monclient: shutdown 2024-09-13T11:42:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 1 -- 172.21.15.5:0/2864246138 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f504c003a60 msgr2=0x7f50601cfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 1 --2- 172.21.15.5:0/2864246138 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f504c003a60 0x7f50601cfa50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5050000c10 tx=0x7f504c005840 comp rx=0 tx=0).stop 2024-09-13T11:42:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f505f59e640 1 -- 172.21.15.5:0/2864246138 reap_dead start 2024-09-13T11:42:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 1 -- 172.21.15.5:0/2864246138 shutdown_connections 2024-09-13T11:42:12.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 1 -- 172.21.15.5:0/2864246138 >> 172.21.15.5:0/2864246138 conn(0x7f50600a6780 msgr2=0x7f50600a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:12.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 1 -- 172.21.15.5:0/2864246138 shutdown_connections 2024-09-13T11:42:12.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:12.923+0000 7f50651ec640 1 -- 172.21.15.5:0/2864246138 wait complete. 2024-09-13T11:42:12.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T11:42:13.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient: get_monmap_and_config 2024-09-13T11:42:13.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient: build_initial_monmap 2024-09-13T11:42:13.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient: monmap: 2024-09-13T11:42:13.057 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:13.057 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.058 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:13.057387+0000 2024-09-13T11:42:13.058 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:13.057387+0000 2024-09-13T11:42:13.058 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:13.058 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.058 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:13.058 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:13.058 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:13.058 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:13.058 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:13.058 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 1 Processor -- start 2024-09-13T11:42:13.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 1 -- start start 2024-09-13T11:42:13.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient: init 2024-09-13T11:42:13.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:13.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-13T11:42:13.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fca14137930 0x7fca1409cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient(hunting): picked mon.noname-c con 0x7fca14137930 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:13.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca14137d10 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:13.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fca1413cd80 0x7fca1413f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient(hunting): picked mon.noname-a con 0x7fca1413cd80 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca140a5a80 con 0x7fca14137d10 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fca140a5fe0 con 0x7fca1413cd80 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fca1409c540 con 0x7fca14137930 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1bba5640 10 monclient(hunting): authenticate will time out at 1136.406217s 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca19140640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fca14137d10 0x7fca141380f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60732/0 (socket says 172.21.15.5:60732) 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca19140640 1 -- 172.21.15.5:0/2571914752 learned_addr learned my addr 172.21.15.5:0/2571914752 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 1 --2- 172.21.15.5:0/2571914752 >> v2:172.21.15.5:3300/0 conn(0x7fca1413cd80 0x7fca1413f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 10 monclient(hunting): get_auth_request con 0x7fca1413cd80 auth_method 0 2024-09-13T11:42:13.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca19941640 1 --2- 172.21.15.5:0/2571914752 >> v2:172.21.15.39:3300/0 conn(0x7fca14137930 0x7fca1409cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca19941640 10 monclient(hunting): get_auth_request con 0x7fca14137930 auth_method 0 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca19941640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca19941640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca19941640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca19941640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca19941640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca19941640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 10 monclient(hunting): handle_auth_done global_id 4644 payload 274 2024-09-13T11:42:13.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 1 -- 172.21.15.5:0/2571914752 >> v2:172.21.15.39:3300/0 conn(0x7fca14137930 msgr2=0x7fca1409cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:13.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 1 --2- 172.21.15.5:0/2571914752 >> v2:172.21.15.39:3300/0 conn(0x7fca14137930 0x7fca1409cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 1 -- 172.21.15.5:0/2571914752 >> v1:172.21.15.5:6789/0 conn(0x7fca14137d10 legacy=0x7fca141380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 10 monclient: _finish_hunting 0 2024-09-13T11:42:13.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 1 monclient: found mon.noname-a 2024-09-13T11:42:13.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:13.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.055+0000 7fca1a142640 1 -- 172.21.15.5:0/2571914752 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1409bef0 con 0x7fca1413cd80 2024-09-13T11:42:13.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1a142640 1 --2- 172.21.15.5:0/2571914752 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca1413cd80 0x7fca1413f170 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fca140a7cb0 tx=0x7fca0c01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2affe9bd62c83a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 1 -- 172.21.15.5:0/2571914752 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0b0070 con 0x7fca1413cd80 2024-09-13T11:42:13.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:13.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 10 monclient: dump: 2024-09-13T11:42:13.069 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.069 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 10 monclient: _finish_auth 0 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 10 monclient: _check_auth_tickets 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 1 -- 172.21.15.5:0/2571914752 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca0c0ab600 con 0x7fca1413cd80 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:13.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 1 -- 172.21.15.5:0/2571914752 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0ab9d0 con 0x7fca1413cd80 2024-09-13T11:42:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 5 monclient: authenticate success, global_id 4644 2024-09-13T11:42:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 4 set_mon_vals no callback set 2024-09-13T11:42:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1893f640 10 monclient: dump: 2024-09-13T11:42:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 10 monclient: shutdown 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 1 -- 172.21.15.5:0/2571914752 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca1413cd80 msgr2=0x7fca1413f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 1 --2- 172.21.15.5:0/2571914752 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca1413cd80 0x7fca1413f170 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fca140a7cb0 tx=0x7fca0c01c340 comp rx=0 tx=0).stop 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 1 -- 172.21.15.5:0/2571914752 shutdown_connections 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 1 --2- 172.21.15.5:0/2571914752 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca1413cd80 0x7fca1413f170 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 1 --2- 172.21.15.5:0/2571914752 >> v2:172.21.15.39:3300/0 conn(0x7fca14137930 0x7fca1409cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 1 -- 172.21.15.5:0/2571914752 >> 172.21.15.5:0/2571914752 conn(0x7fca140a6c70 msgr2=0x7fca140a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 1 -- 172.21.15.5:0/2571914752 shutdown_connections 2024-09-13T11:42:13.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 1 -- 172.21.15.5:0/2571914752 wait complete. 2024-09-13T11:42:13.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 10 monclient: build_initial_monmap 2024-09-13T11:42:13.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 10 monclient: monmap: 2024-09-13T11:42:13.073 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:13.073 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:13.073 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:13.073 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:13.073 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:13.073 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.073 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 1 Processor -- start 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 1 -- start start 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 10 monclient: init 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca14137930 0x7fca141cf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 10 monclient(hunting): picked mon.noname-a con 0x7fca14137930 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.059+0000 7fca1bba5640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca14137d10 0x7fca14136b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient(hunting): picked mon.noname-b con 0x7fca14137d10 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca1413cd80 0x7fca14139d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient(hunting): picked mon.noname-c con 0x7fca1413cd80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fca14142350 con 0x7fca14137930 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fca141421d0 con 0x7fca14137d10 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fca141424d0 con 0x7fca1413cd80 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient(hunting): authenticate will time out at 1136.412104s 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca14137930 0x7fca141cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca14137930 0x7fca141cf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43036/0 (socket says 172.21.15.5:43036) 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 1 -- 172.21.15.5:0/3054161363 learned_addr learned my addr 172.21.15.5:0/3054161363 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 10 monclient(hunting): get_auth_request con 0x7fca14137930 auth_method 0 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1a142640 1 --2- 172.21.15.5:0/3054161363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca1413cd80 0x7fca14139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1a142640 10 monclient(hunting): get_auth_request con 0x7fca1413cd80 auth_method 0 2024-09-13T11:42:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1a142640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1a142640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1a142640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1a142640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1a142640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1a142640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19140640 1 --2- 172.21.15.5:0/3054161363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca14137d10 0x7fca14136b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 10 monclient(hunting): handle_auth_done global_id 4650 payload 824 2024-09-13T11:42:13.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 1 -- 172.21.15.5:0/3054161363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca1413cd80 msgr2=0x7fca14139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 1 --2- 172.21.15.5:0/3054161363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca1413cd80 0x7fca14139d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 1 -- 172.21.15.5:0/3054161363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca14137d10 msgr2=0x7fca14136b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 1 --2- 172.21.15.5:0/3054161363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca14137d10 0x7fca14136b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 10 monclient: _finish_hunting 0 2024-09-13T11:42:13.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 1 monclient: found mon.noname-a 2024-09-13T11:42:13.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:13.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 1 -- 172.21.15.5:0/3054161363 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1413b730 con 0x7fca14137930 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19140640 10 monclient: get_auth_request con 0x7fca14137d10 auth_method 0 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19140640 1 --2- 172.21.15.5:0/3054161363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca14137d10 0x7fca14136b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19941640 1 --2- 172.21.15.5:0/3054161363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca14137930 0x7fca141cf940 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fca0800e930 tx=0x7fca0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 1 -- 172.21.15.5:0/3054161363 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08017070 con 0x7fca14137930 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 10 monclient: dump: 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 10 monclient: _finish_auth 0 2024-09-13T11:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 10 monclient: _check_auth_tickets 2024-09-13T11:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 5 monclient: authenticate success, global_id 4650 2024-09-13T11:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient: _renew_subs 2024-09-13T11:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 1 -- 172.21.15.5:0/3054161363 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca0800ca00 con 0x7fca14137930 2024-09-13T11:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 1 -- 172.21.15.5:0/3054161363 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca140a0e00 con 0x7fca14137930 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 1 -- 172.21.15.5:0/3054161363 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0800cdb0 con 0x7fca14137930 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca07fff640 4 set_mon_vals no callback set 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 10 monclient: dump: 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.080 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient: _renew_subs 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 1 -- 172.21.15.5:0/3054161363 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca14120c80 con 0x7fca14137930 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca1bba5640 1 -- 172.21.15.5:0/3054161363 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca1409cc90 con 0x7fca14137930 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 1 -- 172.21.15.5:0/3054161363 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fca0801a070 con 0x7fca14137930 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 1 --2- 172.21.15.5:0/3054161363 >> v2:172.21.15.39:6812/1340838781 conn(0x7fc9f803b2a0 0x7fc9f803d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca067fc640 1 -- 172.21.15.5:0/3054161363 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7fca0804ee70 con 0x7fca14137930 2024-09-13T11:42:13.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19140640 1 --2- 172.21.15.5:0/3054161363 >> v2:172.21.15.39:6812/1340838781 conn(0x7fc9f803b2a0 0x7fc9f803d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19140640 10 monclient: get_auth_request con 0x7fc9f803b2a0 auth_method 0 2024-09-13T11:42:13.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.063+0000 7fca19140640 1 --2- 172.21.15.5:0/3054161363 >> v2:172.21.15.39:6812/1340838781 conn(0x7fc9f803b2a0 0x7fc9f803d760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fca1413edf0 tx=0x7fca10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.067+0000 7fca067fc640 1 -- 172.21.15.5:0/3054161363 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fca0804dda0 con 0x7fca14137930 2024-09-13T11:42:13.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.067+0000 7fca067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:13.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.067+0000 7fca067fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:13.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.243+0000 7fca1bba5640 1 -- 172.21.15.5:0/3054161363 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fca1413d490 con 0x7fc9f803b2a0 2024-09-13T11:42:13.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.243+0000 7fca067fc640 1 -- 172.21.15.5:0/3054161363 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fca1413d490 con 0x7fc9f803b2a0 2024-09-13T11:42:13.248 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T11:42:13.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 1 -- 172.21.15.5:0/3054161363 >> v2:172.21.15.39:6812/1340838781 conn(0x7fc9f803b2a0 msgr2=0x7fc9f803d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 1 --2- 172.21.15.5:0/3054161363 >> v2:172.21.15.39:6812/1340838781 conn(0x7fc9f803b2a0 0x7fc9f803d760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fca1413edf0 tx=0x7fca10009040 comp rx=0 tx=0).stop 2024-09-13T11:42:13.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 10 monclient: shutdown 2024-09-13T11:42:13.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:13.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 1 -- 172.21.15.5:0/3054161363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca14137930 msgr2=0x7fca141cf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 1 --2- 172.21.15.5:0/3054161363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca14137930 0x7fca141cf940 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fca0800e930 tx=0x7fca0800edf0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 1 -- 172.21.15.5:0/3054161363 shutdown_connections 2024-09-13T11:42:13.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 1 --2- 172.21.15.5:0/3054161363 >> v2:172.21.15.39:6812/1340838781 conn(0x7fc9f803b2a0 0x7fc9f803d760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 1 --2- 172.21.15.5:0/3054161363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca1413cd80 0x7fca14139d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 1 --2- 172.21.15.5:0/3054161363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca14137d10 0x7fca14136b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 1 --2- 172.21.15.5:0/3054161363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca14137930 0x7fca141cf940 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 1 -- 172.21.15.5:0/3054161363 >> 172.21.15.5:0/3054161363 conn(0x7fca140a6c70 msgr2=0x7fca140a9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:13.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 1 -- 172.21.15.5:0/3054161363 shutdown_connections 2024-09-13T11:42:13.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.247+0000 7fca1bba5640 1 -- 172.21.15.5:0/3054161363 wait complete. 2024-09-13T11:42:13.268 INFO:tasks.ceph:Scrubbing osd.3 2024-09-13T11:42:13.268 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:42:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient: get_monmap_and_config 2024-09-13T11:42:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient: build_initial_monmap 2024-09-13T11:42:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient: monmap: 2024-09-13T11:42:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:13.394049+0000 2024-09-13T11:42:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:13.394049+0000 2024-09-13T11:42:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:13.395 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 1 Processor -- start 2024-09-13T11:42:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 1 -- start start 2024-09-13T11:42:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient: init 2024-09-13T11:42:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-13T11:42:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa15c0a7490 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa15c0a5b90 0x7fa15c138150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient(hunting): picked mon.noname-a con 0x7fa15c0a5b90 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa15c13ce90 0x7fa15c13f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient(hunting): picked mon.noname-b con 0x7fa15c13ce90 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa15c0a6290 con 0x7fa15c0a7490 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa15c0a7870 con 0x7fa15c0a5b90 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa15c0a1f10 con 0x7fa15c13ce90 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa1639c0640 10 monclient(hunting): authenticate will time out at 1136.742693s 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa160f5b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa15c0a5b90 0x7fa15c138150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa160f5b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa15c0a5b90 0x7fa15c138150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43048/0 (socket says 172.21.15.5:43048) 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa160f5b640 1 -- 172.21.15.5:0/2093064286 learned_addr learned my addr 172.21.15.5:0/2093064286 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa15c13ce90 0x7fa15c13f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa160f5b640 10 monclient(hunting): get_auth_request con 0x7fa15c0a5b90 auth_method 0 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa160f5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa160f5b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa160f5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 10 monclient(hunting): get_auth_request con 0x7fa15c13ce90 auth_method 0 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa160f5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa160f5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa160f5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 10 monclient(hunting): handle_auth_done global_id 4484 payload 274 2024-09-13T11:42:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 1 -- 172.21.15.5:0/2093064286 >> v2:172.21.15.5:3300/0 conn(0x7fa15c0a5b90 msgr2=0x7fa15c138150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 1 --2- 172.21.15.5:0/2093064286 >> v2:172.21.15.5:3300/0 conn(0x7fa15c0a5b90 0x7fa15c138150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 1 -- 172.21.15.5:0/2093064286 >> v1:172.21.15.39:6789/0 conn(0x7fa15c0a7490 legacy=0x7fa15c137a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 10 monclient: _finish_hunting 0 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 1 monclient: found mon.noname-b 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa161f5d640 1 -- 172.21.15.5:0/2093064286 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa15c0a21f0 con 0x7fa15c13ce90 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.391+0000 7fa160f5b640 1 --2- 172.21.15.5:0/2093064286 >> v2:172.21.15.5:3300/0 conn(0x7fa15c0a5b90 0x7fa15c138150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa161f5d640 1 --2- 172.21.15.5:0/2093064286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa15c13ce90 0x7fa15c13f280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa154009d90 tx=0x7fa15401c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a65b8684ec534d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 1 -- 172.21.15.5:0/2093064286 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1540b2070 con 0x7fa15c13ce90 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 10 monclient: dump: 2024-09-13T11:42:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 10 monclient: _finish_auth 0 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 10 monclient: _check_auth_tickets 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 5 monclient: authenticate success, global_id 4484 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 1 -- 172.21.15.5:0/2093064286 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1540a5850 con 0x7fa15c13ce90 2024-09-13T11:42:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 1 -- 172.21.15.5:0/2093064286 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1540a5c20 con 0x7fa15c13ce90 2024-09-13T11:42:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 4 set_mon_vals no callback set 2024-09-13T11:42:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa14ffff640 10 monclient: dump: 2024-09-13T11:42:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient: shutdown 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 -- 172.21.15.5:0/2093064286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa15c13ce90 msgr2=0x7fa15c13f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 --2- 172.21.15.5:0/2093064286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa15c13ce90 0x7fa15c13f280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa154009d90 tx=0x7fa15401c800 comp rx=0 tx=0).stop 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 -- 172.21.15.5:0/2093064286 shutdown_connections 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 --2- 172.21.15.5:0/2093064286 >> v2:172.21.15.5:3300/0 conn(0x7fa15c0a5b90 0x7fa15c138150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 --2- 172.21.15.5:0/2093064286 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa15c13ce90 0x7fa15c13f280 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 -- 172.21.15.5:0/2093064286 >> 172.21.15.5:0/2093064286 conn(0x7fa15c0aac70 msgr2=0x7fa15c0a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 -- 172.21.15.5:0/2093064286 shutdown_connections 2024-09-13T11:42:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 -- 172.21.15.5:0/2093064286 wait complete. 2024-09-13T11:42:13.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient: build_initial_monmap 2024-09-13T11:42:13.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient: monmap: 2024-09-13T11:42:13.409 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 Processor -- start 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 -- start start 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient: init 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa15c0a5b90 0x7fa15c1cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient(hunting): picked mon.noname-a con 0x7fa15c0a5b90 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa15c0a7490 0x7fa15c136a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient(hunting): picked mon.noname-b con 0x7fa15c0a7490 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa15c13ce90 0x7fa15c09da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient(hunting): picked mon.noname-c con 0x7fa15c13ce90 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa15c142160 con 0x7fa15c0a5b90 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa15c141fe0 con 0x7fa15c0a7490 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fa15c1422e0 con 0x7fa15c13ce90 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:13.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa1639c0640 10 monclient(hunting): authenticate will time out at 1136.747694s 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa16175c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa15c0a5b90 0x7fa15c1cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa161f5d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa15c13ce90 0x7fa15c09da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa161f5d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa15c13ce90 0x7fa15c09da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:55088/0 (socket says 172.21.15.5:55088) 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa16175c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa15c0a5b90 0x7fa15c1cf5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43050/0 (socket says 172.21.15.5:43050) 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa161f5d640 1 -- 172.21.15.5:0/4093068571 learned_addr learned my addr 172.21.15.5:0/4093068571 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa161f5d640 10 monclient(hunting): get_auth_request con 0x7fa15c13ce90 auth_method 0 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa161f5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa161f5d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa161f5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa16175c640 10 monclient(hunting): get_auth_request con 0x7fa15c0a5b90 auth_method 0 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa16175c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa16175c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa16175c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa161f5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa161f5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa161f5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa16175c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa16175c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.395+0000 7fa16175c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa160f5b640 1 --2- 172.21.15.5:0/4093068571 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa15c0a7490 0x7fa15c136a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa160f5b640 10 monclient(hunting): get_auth_request con 0x7fa15c0a7490 auth_method 0 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa160f5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa160f5b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa160f5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa16175c640 10 monclient(hunting): handle_auth_done global_id 4656 payload 824 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa16175c640 1 -- 172.21.15.5:0/4093068571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa15c13ce90 msgr2=0x7fa15c09da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa16175c640 1 --2- 172.21.15.5:0/4093068571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa15c13ce90 0x7fa15c09da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa16175c640 1 -- 172.21.15.5:0/4093068571 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa15c0a7490 msgr2=0x7fa15c136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa16175c640 1 --2- 172.21.15.5:0/4093068571 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa15c0a7490 0x7fa15c136a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa16175c640 10 monclient: _finish_hunting 0 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa16175c640 1 monclient: found mon.noname-a 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa16175c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa16175c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa16175c640 1 -- 172.21.15.5:0/4093068571 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa15c09fbf0 con 0x7fa15c0a5b90 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa161f5d640 1 --2- 172.21.15.5:0/4093068571 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa15c13ce90 0x7fa15c09da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa16175c640 1 --2- 172.21.15.5:0/4093068571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa15c0a5b90 0x7fa15c1cf5f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0a6460 tx=0x7fa15800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 1 -- 172.21.15.5:0/4093068571 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1580052d0 con 0x7fa15c0a5b90 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 10 monclient: dump: 2024-09-13T11:42:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 10 monclient: _finish_auth 0 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 10 monclient: _check_auth_tickets 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 1 -- 172.21.15.5:0/4093068571 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa158005490 con 0x7fa15c0a5b90 2024-09-13T11:42:13.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa1639c0640 5 monclient: authenticate success, global_id 4656 2024-09-13T11:42:13.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa1639c0640 10 monclient: _renew_subs 2024-09-13T11:42:13.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa1639c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa1639c0640 1 -- 172.21.15.5:0/4093068571 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa15c0a02b0 con 0x7fa15c0a5b90 2024-09-13T11:42:13.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:13.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 1 -- 172.21.15.5:0/4093068571 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158007240 con 0x7fa15c0a5b90 2024-09-13T11:42:13.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:13.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 10 monclient: dump: 2024-09-13T11:42:13.416 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.416 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.416 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14f7fe640 4 set_mon_vals no callback set 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa1639c0640 10 monclient: _renew_subs 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa1639c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa1639c0640 1 -- 172.21.15.5:0/4093068571 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa15c13f390 con 0x7fa15c0a5b90 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa1639c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa1639c0640 1 -- 172.21.15.5:0/4093068571 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa15c137a40 con 0x7fa15c0a5b90 2024-09-13T11:42:13.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 1 -- 172.21.15.5:0/4093068571 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fa1580073e0 con 0x7fa15c0a5b90 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 1 --2- 172.21.15.5:0/4093068571 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa14003b2d0 0x7fa14003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 1 -- 172.21.15.5:0/4093068571 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7fa15804e0d0 con 0x7fa15c0a5b90 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 1 --2- 172.21.15.5:0/4093068571 >> v2:172.21.15.39:6804/3188120659 conn(0x7fa140040de0 0x7fa140043200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 1 -- 172.21.15.5:0/4093068571 --> v2:172.21.15.39:6804/3188120659 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa140043870 con 0x7fa140040de0 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 1 -- 172.21.15.5:0/4093068571 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa158055050 con 0x7fa15c0a5b90 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa14dffb640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa160f5b640 1 --2- 172.21.15.5:0/4093068571 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa14003b2d0 0x7fa14003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa160f5b640 10 monclient: get_auth_request con 0x7fa14003b2d0 auth_method 0 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa161f5d640 1 --2- 172.21.15.5:0/4093068571 >> v2:172.21.15.39:6804/3188120659 conn(0x7fa140040de0 0x7fa140043200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa161f5d640 10 monclient: get_auth_request con 0x7fa140040de0 auth_method 0 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa160f5b640 1 --2- 172.21.15.5:0/4093068571 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa14003b2d0 0x7fa14003d790 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0a0900 tx=0x7fa150008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.399+0000 7fa161f5d640 1 --2- 172.21.15.5:0/4093068571 >> v2:172.21.15.39:6804/3188120659 conn(0x7fa140040de0 0x7fa140043200 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.403+0000 7fa14dffb640 1 -- 172.21.15.5:0/4093068571 <== osd.3 v2:172.21.15.39:6804/3188120659 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa140043870 con 0x7fa140040de0 2024-09-13T11:42:13.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 1 -- 172.21.15.5:0/4093068571 --> v2:172.21.15.39:6804/3188120659 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa15c120f90 con 0x7fa140040de0 2024-09-13T11:42:13.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa14dffb640 1 -- 172.21.15.5:0/4093068571 <== osd.3 v2:172.21.15.39:6804/3188120659 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa15c120f90 con 0x7fa140040de0 2024-09-13T11:42:13.419 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:42:13.419 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:42:13.420 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:42:13.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 1 -- 172.21.15.5:0/4093068571 >> v2:172.21.15.39:6804/3188120659 conn(0x7fa140040de0 msgr2=0x7fa140043200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 1 --2- 172.21.15.5:0/4093068571 >> v2:172.21.15.39:6804/3188120659 conn(0x7fa140040de0 0x7fa140043200 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 1 -- 172.21.15.5:0/4093068571 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa14003b2d0 msgr2=0x7fa14003d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 1 --2- 172.21.15.5:0/4093068571 >> v2:172.21.15.39:6812/1340838781 conn(0x7fa14003b2d0 0x7fa14003d790 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0a0900 tx=0x7fa150008040 comp rx=0 tx=0).stop 2024-09-13T11:42:13.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 10 monclient: shutdown 2024-09-13T11:42:13.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:13.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 1 -- 172.21.15.5:0/4093068571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa15c0a5b90 msgr2=0x7fa15c1cf5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 1 --2- 172.21.15.5:0/4093068571 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa15c0a5b90 0x7fa15c1cf5f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0a6460 tx=0x7fa15800c820 comp rx=0 tx=0).stop 2024-09-13T11:42:13.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa161f5d640 1 -- 172.21.15.5:0/4093068571 reap_dead start 2024-09-13T11:42:13.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 1 -- 172.21.15.5:0/4093068571 shutdown_connections 2024-09-13T11:42:13.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 1 -- 172.21.15.5:0/4093068571 >> 172.21.15.5:0/4093068571 conn(0x7fa15c0aac70 msgr2=0x7fa15c13c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:13.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 1 -- 172.21.15.5:0/4093068571 shutdown_connections 2024-09-13T11:42:13.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.415+0000 7fa1639c0640 1 -- 172.21.15.5:0/4093068571 wait complete. 2024-09-13T11:42:13.433 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-13T11:42:13.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.543+0000 7f9a869a0640 10 monclient: get_monmap_and_config 2024-09-13T11:42:13.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.543+0000 7f9a869a0640 10 monclient: build_initial_monmap 2024-09-13T11:42:13.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.543+0000 7f9a869a0640 10 monclient: monmap: 2024-09-13T11:42:13.548 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:13.548 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.548 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:13.547804+0000 2024-09-13T11:42:13.548 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:13.547804+0000 2024-09-13T11:42:13.548 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:13.548 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.548 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:13.549 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:13.549 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:13.549 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:13.549 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:13.549 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.543+0000 7f9a869a0640 1 Processor -- start 2024-09-13T11:42:13.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.543+0000 7f9a869a0640 1 -- start start 2024-09-13T11:42:13.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.543+0000 7f9a869a0640 10 monclient: init 2024-09-13T11:42:13.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:13.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-13T11:42:13.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a80074460 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:13.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9a800a0660 0x7f9a800a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 10 monclient(hunting): picked mon.noname-a con 0x7f9a800a0660 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:13.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a800a1010 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:13.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a800a1b90 con 0x7f9a800a1010 2024-09-13T11:42:13.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8009f400 con 0x7f9a80074460 2024-09-13T11:42:13.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f9a800a4360 con 0x7f9a800a0660 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 10 monclient(hunting): authenticate will time out at 1136.896592s 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9a800a0660 0x7f9a800a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9a800a0660 0x7f9a800a0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43054/0 (socket says 172.21.15.5:43054) 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 1 -- 172.21.15.5:0/3729565516 learned_addr learned my addr 172.21.15.5:0/3729565516 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 10 monclient(hunting): get_auth_request con 0x7f9a800a0660 auth_method 0 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 10 monclient(hunting): handle_auth_done global_id 4659 payload 274 2024-09-13T11:42:13.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 1 -- 172.21.15.5:0/3729565516 >> v1:172.21.15.39:6789/0 conn(0x7f9a80074460 legacy=0x7f9a80074840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 1 -- 172.21.15.5:0/3729565516 >> v1:172.21.15.5:6789/0 conn(0x7f9a800a1010 legacy=0x7f9a800a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 10 monclient: _finish_hunting 0 2024-09-13T11:42:13.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 1 monclient: found mon.noname-a 2024-09-13T11:42:13.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:13.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 1 -- 172.21.15.5:0/3729565516 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8009f6e0 con 0x7f9a800a0660 2024-09-13T11:42:13.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7f7fe640 1 --2- 172.21.15.5:0/3729565516 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a800a0660 0x7f9a800a0ad0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009870 tx=0x7f9a7001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390b6f2cd732e7d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 1 -- 172.21.15.5:0/3729565516 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a700b1070 con 0x7f9a800a0660 2024-09-13T11:42:13.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:13.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 10 monclient: dump: 2024-09-13T11:42:13.558 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.558 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.558 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.558 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.558 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.558 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.558 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 10 monclient: _finish_auth 0 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 10 monclient: _check_auth_tickets 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 1 -- 172.21.15.5:0/3729565516 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a700ac410 con 0x7f9a800a0660 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 1 -- 172.21.15.5:0/3729565516 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a700ac7e0 con 0x7f9a800a0660 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 5 monclient: authenticate success, global_id 4659 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a869a0640 4 set_mon_vals no callback set 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.547+0000 7f9a7effd640 10 monclient: dump: 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient: shutdown 2024-09-13T11:42:13.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 -- 172.21.15.5:0/3729565516 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a800a0660 msgr2=0x7f9a800a0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 --2- 172.21.15.5:0/3729565516 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a800a0660 0x7f9a800a0ad0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009870 tx=0x7f9a7001c3f0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 -- 172.21.15.5:0/3729565516 shutdown_connections 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 --2- 172.21.15.5:0/3729565516 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a800a0660 0x7f9a800a0ad0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 -- 172.21.15.5:0/3729565516 >> 172.21.15.5:0/3729565516 conn(0x7f9a800a4990 msgr2=0x7f9a800a4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 -- 172.21.15.5:0/3729565516 shutdown_connections 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 -- 172.21.15.5:0/3729565516 wait complete. 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient: build_initial_monmap 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient: monmap: 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 Processor -- start 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 -- start start 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient: init 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a80074460 0x7f9a80127b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient(hunting): picked mon.noname-b con 0x7f9a80074460 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a800a0660 0x7f9a8013f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient(hunting): picked mon.noname-c con 0x7f9a800a0660 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a800a1010 0x7f9a801255c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient(hunting): picked mon.noname-a con 0x7f9a800a1010 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9a80143850 con 0x7f9a800a1010 2024-09-13T11:42:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9a801436d0 con 0x7f9a80074460 2024-09-13T11:42:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f9a801439d0 con 0x7f9a800a0660 2024-09-13T11:42:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient(hunting): authenticate will time out at 1136.901802s 2024-09-13T11:42:13.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a800a1010 0x7f9a801255c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7f7fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a800a0660 0x7f9a8013f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a800a1010 0x7f9a801255c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43060/0 (socket says 172.21.15.5:43060) 2024-09-13T11:42:13.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 1 -- 172.21.15.5:0/65681909 learned_addr learned my addr 172.21.15.5:0/65681909 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:13.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 10 monclient(hunting): get_auth_request con 0x7f9a800a1010 auth_method 0 2024-09-13T11:42:13.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7f7fe640 10 monclient(hunting): get_auth_request con 0x7f9a800a0660 auth_method 0 2024-09-13T11:42:13.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7ffff640 1 --2- 172.21.15.5:0/65681909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a80074460 0x7f9a80127b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7ffff640 10 monclient(hunting): get_auth_request con 0x7f9a80074460 auth_method 0 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 10 monclient(hunting): handle_auth_done global_id 4665 payload 824 2024-09-13T11:42:13.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 1 -- 172.21.15.5:0/65681909 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a800a0660 msgr2=0x7f9a8013f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 1 --2- 172.21.15.5:0/65681909 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a800a0660 0x7f9a8013f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 1 -- 172.21.15.5:0/65681909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a80074460 msgr2=0x7f9a80127b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 1 --2- 172.21.15.5:0/65681909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a80074460 0x7f9a80127b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 10 monclient: _finish_hunting 0 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 1 monclient: found mon.noname-a 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 1 -- 172.21.15.5:0/65681909 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a80129390 con 0x7f9a800a1010 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7f7fe640 1 --2- 172.21.15.5:0/65681909 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a800a0660 0x7f9a8013f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a84f3d640 1 --2- 172.21.15.5:0/65681909 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a800a1010 0x7f9a801255c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800c700 tx=0x7f9a7800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 1 -- 172.21.15.5:0/65681909 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78003b80 con 0x7f9a800a1010 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 10 monclient: dump: 2024-09-13T11:42:13.566 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 10 monclient: _finish_auth 0 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 10 monclient: _check_auth_tickets 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 5 monclient: authenticate success, global_id 4665 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 1 -- 172.21.15.5:0/65681909 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a78003d20 con 0x7f9a800a1010 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient: _renew_subs 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 -- 172.21.15.5:0/65681909 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8009f0f0 con 0x7f9a800a1010 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 1 -- 172.21.15.5:0/65681909 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7800aa90 con 0x7f9a800a1010 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 10 monclient: dump: 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient: _renew_subs 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 -- 172.21.15.5:0/65681909 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a80127030 con 0x7f9a800a1010 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7e7fc640 4 set_mon_vals no callback set 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a869a0640 1 -- 172.21.15.5:0/65681909 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a800a0ad0 con 0x7f9a800a1010 2024-09-13T11:42:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 1 -- 172.21.15.5:0/65681909 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f9a7800ac30 con 0x7f9a800a1010 2024-09-13T11:42:13.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.551+0000 7f9a7cff9640 1 --2- 172.21.15.5:0/65681909 >> v2:172.21.15.39:6812/1340838781 conn(0x7f9a6c03b2a0 0x7f9a6c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.555+0000 7f9a7cff9640 1 -- 172.21.15.5:0/65681909 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f9a7804ea80 con 0x7f9a800a1010 2024-09-13T11:42:13.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.555+0000 7f9a7ffff640 1 --2- 172.21.15.5:0/65681909 >> v2:172.21.15.39:6812/1340838781 conn(0x7f9a6c03b2a0 0x7f9a6c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.555+0000 7f9a7ffff640 10 monclient: get_auth_request con 0x7f9a6c03b2a0 auth_method 0 2024-09-13T11:42:13.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.555+0000 7f9a7ffff640 1 --2- 172.21.15.5:0/65681909 >> v2:172.21.15.39:6812/1340838781 conn(0x7f9a6c03b2a0 0x7f9a6c03d760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9a740016f0 tx=0x7f9a74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.559+0000 7f9a7cff9640 1 -- 172.21.15.5:0/65681909 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f9a800a1b90 con 0x7f9a800a1010 2024-09-13T11:42:13.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.559+0000 7f9a7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:13.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.559+0000 7f9a7cff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:13.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 -- 172.21.15.5:0/65681909 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f9a80069ff0 con 0x7f9a6c03b2a0 2024-09-13T11:42:13.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a7cff9640 1 -- 172.21.15.5:0/65681909 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9a80069ff0 con 0x7f9a6c03b2a0 2024-09-13T11:42:13.734 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 3 to deep-scrub 2024-09-13T11:42:13.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 -- 172.21.15.5:0/65681909 >> v2:172.21.15.39:6812/1340838781 conn(0x7f9a6c03b2a0 msgr2=0x7f9a6c03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 --2- 172.21.15.5:0/65681909 >> v2:172.21.15.39:6812/1340838781 conn(0x7f9a6c03b2a0 0x7f9a6c03d760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9a740016f0 tx=0x7f9a74009040 comp rx=0 tx=0).stop 2024-09-13T11:42:13.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 10 monclient: shutdown 2024-09-13T11:42:13.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:13.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 -- 172.21.15.5:0/65681909 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a800a1010 msgr2=0x7f9a801255c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 --2- 172.21.15.5:0/65681909 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a800a1010 0x7f9a801255c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800c700 tx=0x7f9a7800cbc0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 -- 172.21.15.5:0/65681909 shutdown_connections 2024-09-13T11:42:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 --2- 172.21.15.5:0/65681909 >> v2:172.21.15.39:6812/1340838781 conn(0x7f9a6c03b2a0 0x7f9a6c03d760 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 --2- 172.21.15.5:0/65681909 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a800a1010 0x7f9a801255c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 --2- 172.21.15.5:0/65681909 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a800a0660 0x7f9a8013f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 --2- 172.21.15.5:0/65681909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a80074460 0x7f9a80127b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 -- 172.21.15.5:0/65681909 >> 172.21.15.5:0/65681909 conn(0x7f9a800a4990 msgr2=0x7f9a800a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 -- 172.21.15.5:0/65681909 shutdown_connections 2024-09-13T11:42:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.731+0000 7f9a869a0640 1 -- 172.21.15.5:0/65681909 wait complete. 2024-09-13T11:42:13.753 INFO:tasks.ceph:Scrubbing osd.4 2024-09-13T11:42:13.753 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:42:13.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient: get_monmap_and_config 2024-09-13T11:42:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient: build_initial_monmap 2024-09-13T11:42:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient: monmap: 2024-09-13T11:42:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:13.881731+0000 2024-09-13T11:42:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:13.881731+0000 2024-09-13T11:42:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:13.883 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:13.883 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 1 Processor -- start 2024-09-13T11:42:13.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 1 -- start start 2024-09-13T11:42:13.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient: init 2024-09-13T11:42:13.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:13.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-13T11:42:13.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f36200a4ae0 0x7f36200a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient(hunting): picked mon.noname-c con 0x7f36200a4ae0 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:13.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36200a5400 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:13.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f36200a5850 0x7f3620137fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient(hunting): picked mon.noname-a con 0x7f36200a5850 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:13.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36200a1a00 con 0x7f36200a5400 2024-09-13T11:42:13.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f36200a0f50 con 0x7f36200a5850 2024-09-13T11:42:13.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f36200a1fc0 con 0x7f36200a4ae0 2024-09-13T11:42:13.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:13.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f36270a1640 10 monclient(hunting): authenticate will time out at 1137.230688s 2024-09-13T11:42:13.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f36200a5850 0x7f3620137fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f36200a5850 0x7f3620137fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43066/0 (socket says 172.21.15.5:43066) 2024-09-13T11:42:13.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 1 -- 172.21.15.5:0/2648599947 learned_addr learned my addr 172.21.15.5:0/2648599947 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:13.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 10 monclient(hunting): get_auth_request con 0x7f36200a5850 auth_method 0 2024-09-13T11:42:13.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f3624e3d640 1 --2- 172.21.15.5:0/2648599947 >> v2:172.21.15.39:3300/0 conn(0x7f36200a4ae0 0x7f36200a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f3624e3d640 10 monclient(hunting): get_auth_request con 0x7f36200a4ae0 auth_method 0 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f3624e3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f3624e3d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f3624e3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 10 monclient(hunting): handle_auth_done global_id 4668 payload 274 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 1 -- 172.21.15.5:0/2648599947 >> v2:172.21.15.39:3300/0 conn(0x7f36200a4ae0 msgr2=0x7f36200a4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 1 --2- 172.21.15.5:0/2648599947 >> v2:172.21.15.39:3300/0 conn(0x7f36200a4ae0 0x7f36200a4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 1 -- 172.21.15.5:0/2648599947 >> v1:172.21.15.5:6789/0 conn(0x7f36200a5400 legacy=0x7f36201378a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 10 monclient: _finish_hunting 0 2024-09-13T11:42:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 1 monclient: found mon.noname-a 2024-09-13T11:42:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.879+0000 7f362563e640 1 -- 172.21.15.5:0/2648599947 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36200a1ce0 con 0x7f36200a5850 2024-09-13T11:42:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f362563e640 1 --2- 172.21.15.5:0/2648599947 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36200a5850 0x7f3620137fb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f36200a7aa0 tx=0x7f361801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d53bcc3dbcd02f23 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 1 -- 172.21.15.5:0/2648599947 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36180a5680 con 0x7f36200a5850 2024-09-13T11:42:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 10 monclient: dump: 2024-09-13T11:42:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 10 monclient: _finish_auth 0 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 1 -- 172.21.15.5:0/2648599947 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36180a5820 con 0x7f36200a5850 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 1 -- 172.21.15.5:0/2648599947 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36180a5bf0 con 0x7f36200a5850 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 5 monclient: authenticate success, global_id 4668 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 4 set_mon_vals no callback set 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f361f7fe640 10 monclient: dump: 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 10 monclient: shutdown 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 1 -- 172.21.15.5:0/2648599947 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36200a5850 msgr2=0x7f3620137fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 1 --2- 172.21.15.5:0/2648599947 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36200a5850 0x7f3620137fb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f36200a7aa0 tx=0x7f361801c8b0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 1 -- 172.21.15.5:0/2648599947 shutdown_connections 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 1 --2- 172.21.15.5:0/2648599947 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36200a5850 0x7f3620137fb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 1 --2- 172.21.15.5:0/2648599947 >> v2:172.21.15.39:3300/0 conn(0x7f36200a4ae0 0x7f36200a4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 1 -- 172.21.15.5:0/2648599947 >> 172.21.15.5:0/2648599947 conn(0x7f36200aad50 msgr2=0x7f36200a6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 1 -- 172.21.15.5:0/2648599947 shutdown_connections 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 1 -- 172.21.15.5:0/2648599947 wait complete. 2024-09-13T11:42:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 10 monclient: build_initial_monmap 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.883+0000 7f36270a1640 10 monclient: monmap: 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 1 Processor -- start 2024-09-13T11:42:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 1 -- start start 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient: init 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36200a4ae0 0x7f36201373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient(hunting): picked mon.noname-c con 0x7f36200a4ae0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36200a5400 0x7f362013d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient(hunting): picked mon.noname-a con 0x7f36200a5400 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36200a5850 0x7f362013dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient(hunting): picked mon.noname-b con 0x7f36200a5850 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f36200a5ca0 con 0x7f36200a5400 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3620138700 con 0x7f36200a5850 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f36201425a0 con 0x7f36200a4ae0 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient(hunting): authenticate will time out at 1137.236872s 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f3624e3d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36200a4ae0 0x7f36201373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36200a5400 0x7f362013d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36200a5400 0x7f362013d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43068/0 (socket says 172.21.15.5:43068) 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 1 -- 172.21.15.5:0/2258867116 learned_addr learned my addr 172.21.15.5:0/2258867116 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 10 monclient(hunting): get_auth_request con 0x7f36200a5400 auth_method 0 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f3624e3d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36200a4ae0 0x7f36201373d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:55098/0 (socket says 172.21.15.5:55098) 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f3624e3d640 10 monclient(hunting): get_auth_request con 0x7f36200a4ae0 auth_method 0 2024-09-13T11:42:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f3624e3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f3624e3d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f3624e3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f3624e3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f3624e3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f3624e3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f362563e640 1 --2- 172.21.15.5:0/2258867116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36200a5850 0x7f362013dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f362563e640 10 monclient(hunting): get_auth_request con 0x7f36200a5850 auth_method 0 2024-09-13T11:42:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f362563e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f362563e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f362563e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 10 monclient(hunting): handle_auth_done global_id 4671 payload 824 2024-09-13T11:42:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 1 -- 172.21.15.5:0/2258867116 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36200a4ae0 msgr2=0x7f36201373d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 1 --2- 172.21.15.5:0/2258867116 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f36200a4ae0 0x7f36201373d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 1 -- 172.21.15.5:0/2258867116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36200a5850 msgr2=0x7f362013dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 1 --2- 172.21.15.5:0/2258867116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36200a5850 0x7f362013dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 10 monclient: _finish_hunting 0 2024-09-13T11:42:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 1 monclient: found mon.noname-a 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 1 -- 172.21.15.5:0/2258867116 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f362013f070 con 0x7f36200a5400 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361ffff640 1 --2- 172.21.15.5:0/2258867116 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36200a5400 0x7f362013d740 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f360800c500 tx=0x7f360800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 1 -- 172.21.15.5:0/2258867116 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608017070 con 0x7f36200a5400 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 10 monclient: dump: 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 10 monclient: _finish_auth 0 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 10 monclient: _check_auth_tickets 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 1 -- 172.21.15.5:0/2258867116 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36080052d0 con 0x7f36200a5400 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 1 -- 172.21.15.5:0/2258867116 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608012870 con 0x7f36200a5400 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 5 monclient: authenticate success, global_id 4671 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient: _renew_subs 2024-09-13T11:42:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 1 -- 172.21.15.5:0/2258867116 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3620121d80 con 0x7f36200a5400 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361effd640 4 set_mon_vals no callback set 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 10 monclient: dump: 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:13.905 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient: _renew_subs 2024-09-13T11:42:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 1 -- 172.21.15.5:0/2258867116 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3620120730 con 0x7f36200a5400 2024-09-13T11:42:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f36270a1640 1 -- 172.21.15.5:0/2258867116 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f36201c7a80 con 0x7f36200a5400 2024-09-13T11:42:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 1 -- 172.21.15.5:0/2258867116 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f360801a070 con 0x7f36200a5400 2024-09-13T11:42:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 1 --2- 172.21.15.5:0/2258867116 >> v2:172.21.15.39:6812/1340838781 conn(0x7f360003b2d0 0x7f360003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 1 -- 172.21.15.5:0/2258867116 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f360804df10 con 0x7f36200a5400 2024-09-13T11:42:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 1 --2- 172.21.15.5:0/2258867116 >> v2:172.21.15.39:6800/4264535364 conn(0x7f3600040de0 0x7f3600043200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 1 -- 172.21.15.5:0/2258867116 --> v2:172.21.15.39:6800/4264535364 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3600043870 con 0x7f3600040de0 2024-09-13T11:42:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 1 -- 172.21.15.5:0/2258867116 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3608050600 con 0x7f36200a5400 2024-09-13T11:42:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f361d7fa640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-09-13T11:42:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f3624e3d640 1 --2- 172.21.15.5:0/2258867116 >> v2:172.21.15.39:6812/1340838781 conn(0x7f360003b2d0 0x7f360003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f3624e3d640 10 monclient: get_auth_request con 0x7f360003b2d0 auth_method 0 2024-09-13T11:42:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f362563e640 1 --2- 172.21.15.5:0/2258867116 >> v2:172.21.15.39:6800/4264535364 conn(0x7f3600040de0 0x7f3600043200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.887+0000 7f362563e640 10 monclient: get_auth_request con 0x7f3600040de0 auth_method 0 2024-09-13T11:42:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.891+0000 7f362563e640 1 --2- 172.21.15.5:0/2258867116 >> v2:172.21.15.39:6800/4264535364 conn(0x7f3600040de0 0x7f3600043200 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.891+0000 7f3624e3d640 1 --2- 172.21.15.5:0/2258867116 >> v2:172.21.15.39:6812/1340838781 conn(0x7f360003b2d0 0x7f360003d790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3610007a60 tx=0x7f3610008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.891+0000 7f361d7fa640 1 -- 172.21.15.5:0/2258867116 <== osd.4 v2:172.21.15.39:6800/4264535364 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3600043870 con 0x7f3600040de0 2024-09-13T11:42:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 1 -- 172.21.15.5:0/2258867116 --> v2:172.21.15.39:6800/4264535364 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3620137b60 con 0x7f3600040de0 2024-09-13T11:42:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f361d7fa640 1 -- 172.21.15.5:0/2258867116 <== osd.4 v2:172.21.15.39:6800/4264535364 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3620137b60 con 0x7f3600040de0 2024-09-13T11:42:13.909 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:42:13.910 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:42:13.910 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:42:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 1 -- 172.21.15.5:0/2258867116 >> v2:172.21.15.39:6800/4264535364 conn(0x7f3600040de0 msgr2=0x7f3600043200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 1 --2- 172.21.15.5:0/2258867116 >> v2:172.21.15.39:6800/4264535364 conn(0x7f3600040de0 0x7f3600043200 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 1 -- 172.21.15.5:0/2258867116 >> v2:172.21.15.39:6812/1340838781 conn(0x7f360003b2d0 msgr2=0x7f360003d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 1 --2- 172.21.15.5:0/2258867116 >> v2:172.21.15.39:6812/1340838781 conn(0x7f360003b2d0 0x7f360003d790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3610007a60 tx=0x7f3610008040 comp rx=0 tx=0).stop 2024-09-13T11:42:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 10 monclient: shutdown 2024-09-13T11:42:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 1 -- 172.21.15.5:0/2258867116 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36200a5400 msgr2=0x7f362013d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 1 --2- 172.21.15.5:0/2258867116 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f36200a5400 0x7f362013d740 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f360800c500 tx=0x7f360800c9c0 comp rx=0 tx=0).stop 2024-09-13T11:42:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f362563e640 1 -- 172.21.15.5:0/2258867116 reap_dead start 2024-09-13T11:42:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 1 -- 172.21.15.5:0/2258867116 shutdown_connections 2024-09-13T11:42:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 1 -- 172.21.15.5:0/2258867116 >> 172.21.15.5:0/2258867116 conn(0x7f36200aad50 msgr2=0x7f36200a8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:13.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 1 -- 172.21.15.5:0/2258867116 shutdown_connections 2024-09-13T11:42:13.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:13.907+0000 7f36270a1640 1 -- 172.21.15.5:0/2258867116 wait complete. 2024-09-13T11:42:13.926 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-13T11:42:14.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient: get_monmap_and_config 2024-09-13T11:42:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient: build_initial_monmap 2024-09-13T11:42:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient: monmap: 2024-09-13T11:42:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:14.041753+0000 2024-09-13T11:42:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:14.041753+0000 2024-09-13T11:42:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:14.043 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 1 Processor -- start 2024-09-13T11:42:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 1 -- start start 2024-09-13T11:42:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient: init 2024-09-13T11:42:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-13T11:42:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcb30137930 0x7fcb3009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient(hunting): picked mon.noname-a con 0x7fcb30137930 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb30137d10 0x7fcb301380f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient(hunting): picked mon.noname-b con 0x7fcb30137d10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcb300a4a70 0x7fcb300a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient(hunting): picked mon.noname-c con 0x7fcb300a4a70 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fcb300a5fe0 con 0x7fcb30137930 2024-09-13T11:42:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fcb3009c540 con 0x7fcb300a4a70 2024-09-13T11:42:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fcb3009bef0 con 0x7fcb30137d10 2024-09-13T11:42:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb36dfe640 10 monclient(hunting): authenticate will time out at 1137.391080s 2024-09-13T11:42:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcb30137930 0x7fcb3009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb2ffff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb30137d10 0x7fcb301380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb2ffff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb30137d10 0x7fcb301380f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:55112/0 (socket says 172.21.15.5:55112) 2024-09-13T11:42:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcb30137930 0x7fcb3009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43080/0 (socket says 172.21.15.5:43080) 2024-09-13T11:42:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb2ffff640 1 -- 172.21.15.5:0/476346801 learned_addr learned my addr 172.21.15.5:0/476346801 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb2ffff640 10 monclient(hunting): get_auth_request con 0x7fcb30137d10 auth_method 0 2024-09-13T11:42:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 10 monclient(hunting): get_auth_request con 0x7fcb30137930 auth_method 0 2024-09-13T11:42:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb3539b640 1 --2- 172.21.15.5:0/476346801 >> v2:172.21.15.39:3300/0 conn(0x7fcb300a4a70 0x7fcb300a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb3539b640 10 monclient(hunting): get_auth_request con 0x7fcb300a4a70 auth_method 0 2024-09-13T11:42:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb3539b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb3539b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb3539b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 10 monclient(hunting): handle_auth_done global_id 4674 payload 274 2024-09-13T11:42:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 1 -- 172.21.15.5:0/476346801 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb30137d10 msgr2=0x7fcb301380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 1 --2- 172.21.15.5:0/476346801 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb30137d10 0x7fcb301380f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 1 -- 172.21.15.5:0/476346801 >> v2:172.21.15.39:3300/0 conn(0x7fcb300a4a70 msgr2=0x7fcb300a4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 1 --2- 172.21.15.5:0/476346801 >> v2:172.21.15.39:3300/0 conn(0x7fcb300a4a70 0x7fcb300a4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 10 monclient: _finish_hunting 0 2024-09-13T11:42:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 1 monclient: found mon.noname-a 2024-09-13T11:42:14.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:14.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:14.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.039+0000 7fcb34b9a640 1 -- 172.21.15.5:0/476346801 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb3009c1d0 con 0x7fcb30137930 2024-09-13T11:42:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb34b9a640 1 --2- 172.21.15.5:0/476346801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb30137930 0x7fcb3009cc90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcb200097c0 tx=0x7fcb2001c2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06c387b8032dc11 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 1 -- 172.21.15.5:0/476346801 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb200b0070 con 0x7fcb30137930 2024-09-13T11:42:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 10 monclient: dump: 2024-09-13T11:42:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.058 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 10 monclient: _finish_auth 0 2024-09-13T11:42:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:42:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 1 -- 172.21.15.5:0/476346801 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb200ab600 con 0x7fcb30137930 2024-09-13T11:42:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 1 -- 172.21.15.5:0/476346801 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb200ab9d0 con 0x7fcb30137930 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 5 monclient: authenticate success, global_id 4674 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 4 set_mon_vals no callback set 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb2f7fe640 10 monclient: dump: 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.059 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.060 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.060 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.060 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 10 monclient: shutdown 2024-09-13T11:42:14.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:14.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 1 -- 172.21.15.5:0/476346801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb30137930 msgr2=0x7fcb3009cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 1 --2- 172.21.15.5:0/476346801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb30137930 0x7fcb3009cc90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcb200097c0 tx=0x7fcb2001c2c0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 1 -- 172.21.15.5:0/476346801 shutdown_connections 2024-09-13T11:42:14.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 1 --2- 172.21.15.5:0/476346801 >> v2:172.21.15.39:3300/0 conn(0x7fcb300a4a70 0x7fcb300a4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 1 --2- 172.21.15.5:0/476346801 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb30137d10 0x7fcb301380f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 1 --2- 172.21.15.5:0/476346801 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb30137930 0x7fcb3009cc90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 1 -- 172.21.15.5:0/476346801 >> 172.21.15.5:0/476346801 conn(0x7fcb300a6c70 msgr2=0x7fcb300a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:14.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 1 -- 172.21.15.5:0/476346801 shutdown_connections 2024-09-13T11:42:14.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 1 -- 172.21.15.5:0/476346801 wait complete. 2024-09-13T11:42:14.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 10 monclient: build_initial_monmap 2024-09-13T11:42:14.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 10 monclient: monmap: 2024-09-13T11:42:14.061 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:14.061 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 1 Processor -- start 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 1 -- start start 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.043+0000 7fcb36dfe640 10 monclient: init 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:42:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb300a4a70 0x7fcb301cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient(hunting): picked mon.noname-a con 0x7fcb300a4a70 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb30137930 0x7fcb30136a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient(hunting): picked mon.noname-c con 0x7fcb30137930 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb30137d10 0x7fcb3013cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient(hunting): picked mon.noname-b con 0x7fcb30137d10 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fcb30141190 con 0x7fcb300a4a70 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcb30141010 con 0x7fcb30137d10 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fcb30141310 con 0x7fcb30137930 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient(hunting): authenticate will time out at 1137.396320s 2024-09-13T11:42:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb30137930 0x7fcb30136a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb300a4a70 0x7fcb301cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb30137930 0x7fcb30136a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:55116/0 (socket says 172.21.15.5:55116) 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 10 monclient(hunting): get_auth_request con 0x7fcb300a4a70 auth_method 0 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 1 -- 172.21.15.5:0/527702189 learned_addr learned my addr 172.21.15.5:0/527702189 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 10 monclient(hunting): get_auth_request con 0x7fcb30137930 auth_method 0 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 10 monclient(hunting): handle_auth_done global_id 4677 payload 824 2024-09-13T11:42:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 1 -- 172.21.15.5:0/527702189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb30137930 msgr2=0x7fcb30136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 1 --2- 172.21.15.5:0/527702189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb30137930 0x7fcb30136a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 1 -- 172.21.15.5:0/527702189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb30137d10 msgr2=0x7fcb3013cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 1 --2- 172.21.15.5:0/527702189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb30137d10 0x7fcb3013cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 10 monclient: _finish_hunting 0 2024-09-13T11:42:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 1 monclient: found mon.noname-a 2024-09-13T11:42:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 1 -- 172.21.15.5:0/527702189 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb30122400 con 0x7fcb300a4a70 2024-09-13T11:42:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb34b9a640 1 --2- 172.21.15.5:0/527702189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb300a4a70 0x7fcb301cf6f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcb200ab000 tx=0x7fcb200190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 1 -- 172.21.15.5:0/527702189 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb200b8020 con 0x7fcb300a4a70 2024-09-13T11:42:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 10 monclient: dump: 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 10 monclient: _finish_auth 0 2024-09-13T11:42:14.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 10 monclient: _check_auth_tickets 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 1 -- 172.21.15.5:0/527702189 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb200b4410 con 0x7fcb300a4a70 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 1 -- 172.21.15.5:0/527702189 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb200b0040 con 0x7fcb300a4a70 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 5 monclient: authenticate success, global_id 4677 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient: _renew_subs 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 1 -- 172.21.15.5:0/527702189 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb300a0e90 con 0x7fcb300a4a70 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 10 monclient: dump: 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.068 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.069 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.069 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.069 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.069 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.069 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.069 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.069 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.069 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2e7fc640 4 set_mon_vals no callback set 2024-09-13T11:42:14.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient: _renew_subs 2024-09-13T11:42:14.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:14.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 1 -- 172.21.15.5:0/527702189 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb30122120 con 0x7fcb300a4a70 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb36dfe640 1 -- 172.21.15.5:0/527702189 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb3009cc90 con 0x7fcb300a4a70 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 1 -- 172.21.15.5:0/527702189 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fcb200bb020 con 0x7fcb300a4a70 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 1 --2- 172.21.15.5:0/527702189 >> v2:172.21.15.39:6812/1340838781 conn(0x7fcb1c03b2a0 0x7fcb1c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2d7fa640 1 -- 172.21.15.5:0/527702189 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7fcb200e8830 con 0x7fcb300a4a70 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 1 --2- 172.21.15.5:0/527702189 >> v2:172.21.15.39:6812/1340838781 conn(0x7fcb1c03b2a0 0x7fcb1c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 10 monclient: get_auth_request con 0x7fcb1c03b2a0 auth_method 0 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.047+0000 7fcb2ffff640 1 --2- 172.21.15.5:0/527702189 >> v2:172.21.15.39:6812/1340838781 conn(0x7fcb1c03b2a0 0x7fcb1c03d760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcb3013ecb0 tx=0x7fcb24009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.051+0000 7fcb2d7fa640 1 -- 172.21.15.5:0/527702189 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fcb200c2030 con 0x7fcb300a4a70 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.051+0000 7fcb2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.051+0000 7fcb2d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.223+0000 7fcb36dfe640 1 -- 172.21.15.5:0/527702189 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fcb301d0570 con 0x7fcb1c03b2a0 2024-09-13T11:42:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.223+0000 7fcb2d7fa640 1 -- 172.21.15.5:0/527702189 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fcb301d0570 con 0x7fcb1c03b2a0 2024-09-13T11:42:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 4 to deep-scrub 2024-09-13T11:42:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 1 -- 172.21.15.5:0/527702189 >> v2:172.21.15.39:6812/1340838781 conn(0x7fcb1c03b2a0 msgr2=0x7fcb1c03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 1 --2- 172.21.15.5:0/527702189 >> v2:172.21.15.39:6812/1340838781 conn(0x7fcb1c03b2a0 0x7fcb1c03d760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcb3013ecb0 tx=0x7fcb24009240 comp rx=0 tx=0).stop 2024-09-13T11:42:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 10 monclient: shutdown 2024-09-13T11:42:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 1 -- 172.21.15.5:0/527702189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb300a4a70 msgr2=0x7fcb301cf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 1 --2- 172.21.15.5:0/527702189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb300a4a70 0x7fcb301cf6f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcb200ab000 tx=0x7fcb200190c0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 1 -- 172.21.15.5:0/527702189 shutdown_connections 2024-09-13T11:42:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 1 --2- 172.21.15.5:0/527702189 >> v2:172.21.15.39:6812/1340838781 conn(0x7fcb1c03b2a0 0x7fcb1c03d760 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 1 --2- 172.21.15.5:0/527702189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb30137d10 0x7fcb3013cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 1 --2- 172.21.15.5:0/527702189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb30137930 0x7fcb30136a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 1 --2- 172.21.15.5:0/527702189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb300a4a70 0x7fcb301cf6f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 1 -- 172.21.15.5:0/527702189 >> 172.21.15.5:0/527702189 conn(0x7fcb300a6c70 msgr2=0x7fcb3009fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 1 -- 172.21.15.5:0/527702189 shutdown_connections 2024-09-13T11:42:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.227+0000 7fcb36dfe640 1 -- 172.21.15.5:0/527702189 wait complete. 2024-09-13T11:42:14.247 INFO:tasks.ceph:Scrubbing osd.5 2024-09-13T11:42:14.247 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:42:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient: get_monmap_and_config 2024-09-13T11:42:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient: build_initial_monmap 2024-09-13T11:42:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient: monmap: 2024-09-13T11:42:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:14.377574+0000 2024-09-13T11:42:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:14.377574+0000 2024-09-13T11:42:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:14.378 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 1 Processor -- start 2024-09-13T11:42:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 1 -- start start 2024-09-13T11:42:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient: init 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f43d80a7490 0x7f43d8137a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient(hunting): picked mon.noname-a con 0x7f43d80a7490 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43d8137f80 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f43d8138390 0x7f43d80a5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient(hunting): picked mon.noname-c con 0x7f43d8138390 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43d80a6290 con 0x7f43d8137f80 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f43d80a7870 con 0x7f43d80a7490 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f43d80a1f10 con 0x7f43d8138390 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43e039e640 10 monclient(hunting): authenticate will time out at 1137.725941s 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f43d80a7490 0x7f43d8137a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f43d80a7490 0x7f43d8137a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43092/0 (socket says 172.21.15.5:43092) 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 1 -- 172.21.15.5:0/3252824580 learned_addr learned my addr 172.21.15.5:0/3252824580 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 10 monclient(hunting): get_auth_request con 0x7f43d80a7490 auth_method 0 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de93b640 1 --2- 172.21.15.5:0/3252824580 >> v2:172.21.15.39:3300/0 conn(0x7f43d8138390 0x7f43d80a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de93b640 10 monclient(hunting): get_auth_request con 0x7f43d8138390 auth_method 0 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de93b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de93b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de93b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 10 monclient(hunting): handle_auth_done global_id 4680 payload 274 2024-09-13T11:42:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 1 -- 172.21.15.5:0/3252824580 >> v2:172.21.15.39:3300/0 conn(0x7f43d8138390 msgr2=0x7f43d80a5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 1 --2- 172.21.15.5:0/3252824580 >> v2:172.21.15.39:3300/0 conn(0x7f43d8138390 0x7f43d80a5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 1 -- 172.21.15.5:0/3252824580 >> v1:172.21.15.5:6789/0 conn(0x7f43d8137f80 legacy=0x7f43d80a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 10 monclient: _finish_hunting 0 2024-09-13T11:42:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 1 monclient: found mon.noname-a 2024-09-13T11:42:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.375+0000 7f43de13a640 1 -- 172.21.15.5:0/3252824580 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d80a21f0 con 0x7f43d80a7490 2024-09-13T11:42:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 1 --2- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43d80a7490 0x7f43d8137a40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f43d80a6110 tx=0x7f43cc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1dccac30597bcb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 1 -- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43d80a7490 msgr2=0x7f43d8137a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-13T11:42:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 1 -- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43d80a7490 msgr2=0x7f43d8137a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T11:42:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 1 --2- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43d80a7490 0x7f43d8137a40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f43d80a6110 tx=0x7f43cc01c3e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T11:42:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 1 --2- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43d80a7490 0x7f43d8137a40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f43d80a6110 tx=0x7f43cc01c3e0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient: ms_handle_reset current mon [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 1 -- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43d80a7490 msgr2=0x7f43d8137a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T11:42:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 1 --2- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43d80a7490 0x7f43d8137a40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient: _add_conns ranks=[2,1] 2024-09-13T11:42:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 1 --2- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43c0003510 0x7f43c0005960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient(hunting): picked mon.noname-b con 0x7f43c0003510 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43c0006230 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 1 -- 172.21.15.5:0/3252824580 --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43c000a1a0 con 0x7f43c0006230 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 1 -- 172.21.15.5:0/3252824580 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f43c000b3f0 con 0x7f43c0003510 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 1 --2- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43c0003510 0x7f43c0005960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 10 monclient(hunting): get_auth_request con 0x7f43c0003510 auth_method 0 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T11:42:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 10 monclient(hunting): handle_auth_done global_id 4680 payload 293 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 1 -- 172.21.15.5:0/3252824580 >> v1:172.21.15.39:6789/0 conn(0x7f43c0006230 legacy=0x7f43c00086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 10 monclient: _finish_hunting 0 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 1 monclient: found mon.noname-b 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 1 -- 172.21.15.5:0/3252824580 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c000b6b0 con 0x7f43c0003510 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de13a640 1 --2- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43c0003510 0x7f43c0005960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f43d8069290 tx=0x7f43cc009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=844f6a4c625bf13a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 1 -- 172.21.15.5:0/3252824580 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc0b0070 con 0x7f43c0003510 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:42:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient: dump: 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient: _finish_auth 0 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient: _check_auth_tickets 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 5 monclient: authenticate success, global_id 4680 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 1 -- 172.21.15.5:0/3252824580 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43cc0bb070 con 0x7f43c0003510 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 4 set_mon_vals no callback set 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 1 -- 172.21.15.5:0/3252824580 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc003a60 con 0x7f43c0003510 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43dd138640 10 monclient: dump: 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient: shutdown 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 -- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43c0003510 msgr2=0x7f43c0005960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 --2- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43c0003510 0x7f43c0005960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f43d8069290 tx=0x7f43cc009240 comp rx=0 tx=0).stop 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43de93b640 1 -- 172.21.15.5:0/3252824580 reap_dead start 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 -- 172.21.15.5:0/3252824580 shutdown_connections 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 --2- 172.21.15.5:0/3252824580 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43d80a7490 0x7f43d8137a40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 -- 172.21.15.5:0/3252824580 >> 172.21.15.5:0/3252824580 conn(0x7f43d80aac70 msgr2=0x7f43d80a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 -- 172.21.15.5:0/3252824580 shutdown_connections 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 -- 172.21.15.5:0/3252824580 wait complete. 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient: build_initial_monmap 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient: monmap: 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:14.397 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 Processor -- start 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 -- start start 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient: init 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43d80a7490 0x7f43d8136a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient(hunting): picked mon.noname-a con 0x7f43d80a7490 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43c0003510 0x7f43d813cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient(hunting): picked mon.noname-b con 0x7f43c0003510 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43d8136f80 0x7f43d813d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient(hunting): picked mon.noname-c con 0x7f43d8136f80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f43d8142250 con 0x7f43d80a7490 2024-09-13T11:42:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f43d81420d0 con 0x7f43c0003510 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f43d81423d0 con 0x7f43d8136f80 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.379+0000 7f43e039e640 10 monclient(hunting): authenticate will time out at 1137.731984s 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43d8136f80 0x7f43d813d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43d8136f80 0x7f43d813d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:55136/0 (socket says 172.21.15.5:55136) 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 1 -- 172.21.15.5:0/555705245 learned_addr learned my addr 172.21.15.5:0/555705245 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 10 monclient(hunting): get_auth_request con 0x7f43d8136f80 auth_method 0 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de13a640 1 --2- 172.21.15.5:0/555705245 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43d80a7490 0x7f43d8136a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de13a640 10 monclient(hunting): get_auth_request con 0x7f43d80a7490 auth_method 0 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de13a640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43dd939640 1 --2- 172.21.15.5:0/555705245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43c0003510 0x7f43d813cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43dd939640 10 monclient(hunting): get_auth_request con 0x7f43c0003510 auth_method 0 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43dd939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43dd939640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43dd939640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 10 monclient(hunting): handle_auth_done global_id 4502 payload 824 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 1 -- 172.21.15.5:0/555705245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43c0003510 msgr2=0x7f43d813cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 1 --2- 172.21.15.5:0/555705245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43c0003510 0x7f43d813cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 1 -- 172.21.15.5:0/555705245 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43d80a7490 msgr2=0x7f43d8136a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 1 --2- 172.21.15.5:0/555705245 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43d80a7490 0x7f43d8136a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 10 monclient: _finish_hunting 0 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 1 monclient: found mon.noname-c 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 1 -- 172.21.15.5:0/555705245 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d81213f0 con 0x7f43d8136f80 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de93b640 1 --2- 172.21.15.5:0/555705245 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43d8136f80 0x7f43d813d1e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f43d80a61c0 tx=0x7f43d000d8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 1 -- 172.21.15.5:0/555705245 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d0004020 con 0x7f43d8136f80 2024-09-13T11:42:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 10 monclient: dump: 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 10 monclient: _finish_auth 0 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 10 monclient: _check_auth_tickets 2024-09-13T11:42:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 1 -- 172.21.15.5:0/555705245 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d00041e0 con 0x7f43d8136f80 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43e039e640 5 monclient: authenticate success, global_id 4502 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43e039e640 10 monclient: _renew_subs 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43e039e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43e039e640 1 -- 172.21.15.5:0/555705245 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d813f360 con 0x7f43d8136f80 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 1 -- 172.21.15.5:0/555705245 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d00043d0 con 0x7f43d8136f80 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 10 monclient: dump: 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43e039e640 10 monclient: _renew_subs 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43dc937640 4 set_mon_vals no callback set 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43e039e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43e039e640 1 -- 172.21.15.5:0/555705245 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d813e670 con 0x7f43d8136f80 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43e039e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43e039e640 1 -- 172.21.15.5:0/555705245 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f43d80a8b50 con 0x7f43d8136f80 2024-09-13T11:42:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 1 -- 172.21.15.5:0/555705245 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f43d001a020 con 0x7f43d8136f80 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 1 --2- 172.21.15.5:0/555705245 >> v2:172.21.15.39:6812/1340838781 conn(0x7f43c403b2d0 0x7f43c403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 1 -- 172.21.15.5:0/555705245 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f43d004e5b0 con 0x7f43d8136f80 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 1 --2- 172.21.15.5:0/555705245 >> v2:172.21.15.39:6808/1185247880 conn(0x7f43c4040de0 0x7f43c4043200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 1 -- 172.21.15.5:0/555705245 --> v2:172.21.15.39:6808/1185247880 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43c4043870 con 0x7f43c4040de0 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 1 -- 172.21.15.5:0/555705245 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f43d0055050 con 0x7f43d8136f80 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43d6ffd640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de13a640 1 --2- 172.21.15.5:0/555705245 >> v2:172.21.15.39:6812/1340838781 conn(0x7f43c403b2d0 0x7f43c403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de13a640 10 monclient: get_auth_request con 0x7f43c403b2d0 auth_method 0 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43dd939640 1 --2- 172.21.15.5:0/555705245 >> v2:172.21.15.39:6808/1185247880 conn(0x7f43c4040de0 0x7f43c4043200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43dd939640 10 monclient: get_auth_request con 0x7f43c4040de0 auth_method 0 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43de13a640 1 --2- 172.21.15.5:0/555705245 >> v2:172.21.15.39:6812/1340838781 conn(0x7f43c403b2d0 0x7f43c403d790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f43cc0b7000 tx=0x7f43cc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.383+0000 7f43dd939640 1 --2- 172.21.15.5:0/555705245 >> v2:172.21.15.39:6808/1185247880 conn(0x7f43c4040de0 0x7f43c4043200 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.387+0000 7f43d6ffd640 1 -- 172.21.15.5:0/555705245 <== osd.5 v2:172.21.15.39:6808/1185247880 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f43c4043870 con 0x7f43c4040de0 2024-09-13T11:42:14.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 1 -- 172.21.15.5:0/555705245 --> v2:172.21.15.39:6808/1185247880 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f43d8137a40 con 0x7f43c4040de0 2024-09-13T11:42:14.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43d6ffd640 1 -- 172.21.15.5:0/555705245 <== osd.5 v2:172.21.15.39:6808/1185247880 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f43d8137a40 con 0x7f43c4040de0 2024-09-13T11:42:14.406 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T11:42:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:42:14.407 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T11:42:14.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 1 -- 172.21.15.5:0/555705245 >> v2:172.21.15.39:6808/1185247880 conn(0x7f43c4040de0 msgr2=0x7f43c4043200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 1 --2- 172.21.15.5:0/555705245 >> v2:172.21.15.39:6808/1185247880 conn(0x7f43c4040de0 0x7f43c4043200 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 1 -- 172.21.15.5:0/555705245 >> v2:172.21.15.39:6812/1340838781 conn(0x7f43c403b2d0 msgr2=0x7f43c403d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 1 --2- 172.21.15.5:0/555705245 >> v2:172.21.15.39:6812/1340838781 conn(0x7f43c403b2d0 0x7f43c403d790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f43cc0b7000 tx=0x7f43cc008030 comp rx=0 tx=0).stop 2024-09-13T11:42:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 10 monclient: shutdown 2024-09-13T11:42:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 1 -- 172.21.15.5:0/555705245 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43d8136f80 msgr2=0x7f43d813d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 1 --2- 172.21.15.5:0/555705245 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43d8136f80 0x7f43d813d1e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f43d80a61c0 tx=0x7f43d000d8f0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43de93b640 1 -- 172.21.15.5:0/555705245 reap_dead start 2024-09-13T11:42:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 1 -- 172.21.15.5:0/555705245 shutdown_connections 2024-09-13T11:42:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 1 -- 172.21.15.5:0/555705245 >> 172.21.15.5:0/555705245 conn(0x7f43d80aac70 msgr2=0x7f43d80a04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:14.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 1 -- 172.21.15.5:0/555705245 shutdown_connections 2024-09-13T11:42:14.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.399+0000 7f43e039e640 1 -- 172.21.15.5:0/555705245 wait complete. 2024-09-13T11:42:14.422 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-13T11:42:14.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.531+0000 7f039fcfa640 10 monclient: get_monmap_and_config 2024-09-13T11:42:14.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.531+0000 7f039fcfa640 10 monclient: build_initial_monmap 2024-09-13T11:42:14.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.531+0000 7f039fcfa640 10 monclient: monmap: 2024-09-13T11:42:14.537 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:14.537 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.537 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:14.536692+0000 2024-09-13T11:42:14.538 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:14.536692+0000 2024-09-13T11:42:14.538 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:14.539 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.539 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:14.539 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:14.539 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:14.539 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:14.539 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:14.539 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 1 Processor -- start 2024-09-13T11:42:14.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 1 -- start start 2024-09-13T11:42:14.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 10 monclient: init 2024-09-13T11:42:14.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:14.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-13T11:42:14.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4be0 0x7f039809cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 10 monclient(hunting): picked mon.noname-b con 0x7f03980a4be0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:14.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f03980a4fc0 0x7f03980a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 10 monclient(hunting): picked mon.noname-c con 0x7f03980a4fc0 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:14.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03981379a0 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:14.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03980a6510 con 0x7f03981379a0 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f039809ba40 con 0x7f03980a4fc0 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f03980a6a80 con 0x7f03980a4be0 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039fcfa640 10 monclient(hunting): authenticate will time out at 1137.885333s 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4be0 0x7f039809cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4be0 0x7f039809cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:55138/0 (socket says 172.21.15.5:55138) 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 1 -- 172.21.15.5:0/3897792001 learned_addr learned my addr 172.21.15.5:0/3897792001 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 10 monclient(hunting): get_auth_request con 0x7f03980a4be0 auth_method 0 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039d295640 1 --2- 172.21.15.5:0/3897792001 >> v2:172.21.15.39:3300/0 conn(0x7f03980a4fc0 0x7f03980a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039d295640 10 monclient(hunting): get_auth_request con 0x7f03980a4fc0 auth_method 0 2024-09-13T11:42:14.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039d295640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039d295640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039d295640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 10 monclient(hunting): handle_auth_done global_id 4505 payload 274 2024-09-13T11:42:14.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 1 -- 172.21.15.5:0/3897792001 >> v2:172.21.15.39:3300/0 conn(0x7f03980a4fc0 msgr2=0x7f03980a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:14.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 1 --2- 172.21.15.5:0/3897792001 >> v2:172.21.15.39:3300/0 conn(0x7f03980a4fc0 0x7f03980a53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 1 -- 172.21.15.5:0/3897792001 >> v1:172.21.15.39:6789/0 conn(0x7f03981379a0 legacy=0x7f0398137e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 10 monclient: _finish_hunting 0 2024-09-13T11:42:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 1 monclient: found mon.noname-b 2024-09-13T11:42:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:42:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.535+0000 7f039da96640 1 -- 172.21.15.5:0/3897792001 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f039809be50 con 0x7f03980a4be0 2024-09-13T11:42:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039da96640 1 --2- 172.21.15.5:0/3897792001 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4be0 0x7f039809cb10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0388009870 tx=0x7f038801c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7924445393743f24 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 1 -- 172.21.15.5:0/3897792001 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03880b0070 con 0x7f03980a4be0 2024-09-13T11:42:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:42:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 10 monclient: dump: 2024-09-13T11:42:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.553 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 10 monclient: _finish_auth 0 2024-09-13T11:42:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 10 monclient: _check_auth_tickets 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 1 -- 172.21.15.5:0/3897792001 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03880ab600 con 0x7f03980a4be0 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 1 -- 172.21.15.5:0/3897792001 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03880ab9d0 con 0x7f03980a4be0 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039ca94640 10 monclient: dump: 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.555 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 5 monclient: authenticate success, global_id 4505 2024-09-13T11:42:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 4 set_mon_vals no callback set 2024-09-13T11:42:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient: shutdown 2024-09-13T11:42:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 -- 172.21.15.5:0/3897792001 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4be0 msgr2=0x7f039809cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 --2- 172.21.15.5:0/3897792001 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4be0 0x7f039809cb10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0388009870 tx=0x7f038801c2d0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 -- 172.21.15.5:0/3897792001 shutdown_connections 2024-09-13T11:42:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 --2- 172.21.15.5:0/3897792001 >> v2:172.21.15.39:3300/0 conn(0x7f03980a4fc0 0x7f03980a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 --2- 172.21.15.5:0/3897792001 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4be0 0x7f039809cb10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 -- 172.21.15.5:0/3897792001 >> 172.21.15.5:0/3897792001 conn(0x7f03980a6ce0 msgr2=0x7f03980a70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 -- 172.21.15.5:0/3897792001 shutdown_connections 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 -- 172.21.15.5:0/3897792001 wait complete. 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient: build_initial_monmap 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient: monmap: 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 Processor -- start 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 -- start start 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient: init 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03980a4be0 0x7f03981cff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient(hunting): picked mon.noname-a con 0x7f03980a4be0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4fc0 0x7f03981d0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient(hunting): picked mon.noname-c con 0x7f03980a4fc0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03981379a0 0x7f039813cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient(hunting): picked mon.noname-b con 0x7f03981379a0 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0398142820 con 0x7f03980a4be0 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f03981426a0 con 0x7f03981379a0 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f03981429a0 con 0x7f03980a4fc0 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039fcfa640 10 monclient(hunting): authenticate will time out at 1137.891396s 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4fc0 0x7f03981d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039da96640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03980a4be0 0x7f03981cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4fc0 0x7f03981d0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:55152/0 (socket says 172.21.15.5:55152) 2024-09-13T11:42:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 1 -- 172.21.15.5:0/2119167529 learned_addr learned my addr 172.21.15.5:0/2119167529 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 10 monclient(hunting): get_auth_request con 0x7f03980a4fc0 auth_method 0 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039da96640 10 monclient(hunting): get_auth_request con 0x7f03980a4be0 auth_method 0 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039da96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039da96640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039da96640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039da96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039da96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039da96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039e297640 1 --2- 172.21.15.5:0/2119167529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03981379a0 0x7f039813cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039e297640 10 monclient(hunting): get_auth_request con 0x7f03981379a0 auth_method 0 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039e297640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039e297640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039e297640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 10 monclient(hunting): handle_auth_done global_id 4508 payload 824 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 1 -- 172.21.15.5:0/2119167529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03981379a0 msgr2=0x7f039813cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 1 --2- 172.21.15.5:0/2119167529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03981379a0 0x7f039813cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 1 -- 172.21.15.5:0/2119167529 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03980a4be0 msgr2=0x7f03981cff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 1 --2- 172.21.15.5:0/2119167529 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03980a4be0 0x7f03981cff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 10 monclient: _finish_hunting 0 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 1 monclient: found mon.noname-c 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039d295640 1 -- 172.21.15.5:0/2119167529 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03980a13d0 con 0x7f03980a4fc0 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.539+0000 7f039da96640 1 --2- 172.21.15.5:0/2119167529 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03980a4be0 0x7f03981cff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039d295640 1 --2- 172.21.15.5:0/2119167529 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4fc0 0x7f03981d0730 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f039400d800 tx=0x7f039400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 1 -- 172.21.15.5:0/2119167529 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039400b660 con 0x7f03980a4fc0 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 10 monclient: dump: 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 10 monclient: _finish_auth 0 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 10 monclient: _check_auth_tickets 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 1 -- 172.21.15.5:0/2119167529 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0394019070 con 0x7f03980a4fc0 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 1 -- 172.21.15.5:0/2119167529 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039400ba00 con 0x7f03980a4fc0 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039fcfa640 5 monclient: authenticate success, global_id 4508 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039fcfa640 10 monclient: _renew_subs 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039fcfa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039fcfa640 1 -- 172.21.15.5:0/2119167529 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03981216f0 con 0x7f03980a4fc0 2024-09-13T11:42:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 10 monclient: dump: 2024-09-13T11:42:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038ffff640 4 set_mon_vals no callback set 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039fcfa640 10 monclient: _renew_subs 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039fcfa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039fcfa640 1 -- 172.21.15.5:0/2119167529 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f039813f540 con 0x7f03980a4fc0 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039fcfa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039fcfa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039fcfa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039fcfa640 1 -- 172.21.15.5:0/2119167529 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f039809cb10 con 0x7f03980a4fc0 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 1 -- 172.21.15.5:0/2119167529 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f039400bba0 con 0x7f03980a4fc0 2024-09-13T11:42:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 1 --2- 172.21.15.5:0/2119167529 >> v2:172.21.15.39:6812/1340838781 conn(0x7f037c03b2a0 0x7f037c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f038e7fc640 1 -- 172.21.15.5:0/2119167529 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f039404f1d0 con 0x7f03980a4fc0 2024-09-13T11:42:14.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039da96640 1 --2- 172.21.15.5:0/2119167529 >> v2:172.21.15.39:6812/1340838781 conn(0x7f037c03b2a0 0x7f037c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039da96640 10 monclient: get_auth_request con 0x7f037c03b2a0 auth_method 0 2024-09-13T11:42:14.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.543+0000 7f039da96640 1 --2- 172.21.15.5:0/2119167529 >> v2:172.21.15.39:6812/1340838781 conn(0x7f037c03b2a0 0x7f037c03d760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f03880ad000 tx=0x7f0388008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.547+0000 7f038e7fc640 1 -- 172.21.15.5:0/2119167529 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f03980a13d0 con 0x7f03980a4fc0 2024-09-13T11:42:14.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.547+0000 7f038e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:14.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.547+0000 7f038e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:14.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.715+0000 7f039fcfa640 1 -- 172.21.15.5:0/2119167529 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f039806cc70 con 0x7f037c03b2a0 2024-09-13T11:42:14.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.715+0000 7f038e7fc640 1 -- 172.21.15.5:0/2119167529 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f039806cc70 con 0x7f037c03b2a0 2024-09-13T11:42:14.720 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 5 to deep-scrub 2024-09-13T11:42:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 1 -- 172.21.15.5:0/2119167529 >> v2:172.21.15.39:6812/1340838781 conn(0x7f037c03b2a0 msgr2=0x7f037c03d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 1 --2- 172.21.15.5:0/2119167529 >> v2:172.21.15.39:6812/1340838781 conn(0x7f037c03b2a0 0x7f037c03d760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f03880ad000 tx=0x7f0388008040 comp rx=0 tx=0).stop 2024-09-13T11:42:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 10 monclient: shutdown 2024-09-13T11:42:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 1 -- 172.21.15.5:0/2119167529 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4fc0 msgr2=0x7f03981d0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 1 --2- 172.21.15.5:0/2119167529 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4fc0 0x7f03981d0730 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f039400d800 tx=0x7f039400dcc0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 1 -- 172.21.15.5:0/2119167529 shutdown_connections 2024-09-13T11:42:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 1 --2- 172.21.15.5:0/2119167529 >> v2:172.21.15.39:6812/1340838781 conn(0x7f037c03b2a0 0x7f037c03d760 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 1 --2- 172.21.15.5:0/2119167529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03981379a0 0x7f039813cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 1 --2- 172.21.15.5:0/2119167529 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f03980a4fc0 0x7f03981d0730 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 1 --2- 172.21.15.5:0/2119167529 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f03980a4be0 0x7f03981cff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 1 -- 172.21.15.5:0/2119167529 >> 172.21.15.5:0/2119167529 conn(0x7f03980a6ce0 msgr2=0x7f03980a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:14.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 1 -- 172.21.15.5:0/2119167529 shutdown_connections 2024-09-13T11:42:14.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.719+0000 7f039fcfa640 1 -- 172.21.15.5:0/2119167529 wait complete. 2024-09-13T11:42:14.740 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:42:14.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient: get_monmap_and_config 2024-09-13T11:42:14.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient: build_initial_monmap 2024-09-13T11:42:14.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient: monmap: 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:14.858982+0000 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:14.858982+0000 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 1 Processor -- start 2024-09-13T11:42:14.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 1 -- start start 2024-09-13T11:42:14.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient: init 2024-09-13T11:42:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-13T11:42:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f080a0ea0 addr v1:172.21.15.5:6789/0 2024-09-13T11:42:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 0x7f8f0813a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient(hunting): picked mon.noname-b con 0x7f8f080a1ae0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8f0813ad30 0x7f8f0813d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient(hunting): picked mon.noname-a con 0x7f8f0813ad30 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f0806f010 con 0x7f8f080a0ea0 2024-09-13T11:42:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f8f0806f640 con 0x7f8f0813ad30 2024-09-13T11:42:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f8f080a0000 con 0x7f8f080a1ae0 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0edcb640 10 monclient(hunting): authenticate will time out at 1138.207633s 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0cb67640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f8f080a0ea0 0x7f8f080a1280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60752/0 (socket says 172.21.15.5:60752) 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f07fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 0x7f8f0813a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0cb67640 1 -- 172.21.15.5:0/2881264421 learned_addr learned my addr 172.21.15.5:0/2881264421 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0d368640 1 --2- 172.21.15.5:0/2881264421 >> v2:172.21.15.5:3300/0 conn(0x7f8f0813ad30 0x7f8f0813d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f07fff640 10 monclient(hunting): get_auth_request con 0x7f8f080a1ae0 auth_method 0 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f07fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0d368640 10 monclient(hunting): get_auth_request con 0x7f8f0813ad30 auth_method 0 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0d368640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0d368640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0d368640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0d368640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.855+0000 7f8f0d368640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0d368640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0d368640 10 monclient(hunting): handle_auth_done global_id 4689 payload 274 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0d368640 1 -- 172.21.15.5:0/2881264421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 msgr2=0x7f8f0813a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0d368640 1 --2- 172.21.15.5:0/2881264421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 0x7f8f0813a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0d368640 1 -- 172.21.15.5:0/2881264421 >> v1:172.21.15.5:6789/0 conn(0x7f8f080a0ea0 legacy=0x7f8f080a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0d368640 10 monclient: _finish_hunting 0 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0d368640 1 monclient: found mon.noname-a 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0d368640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0d368640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0d368640 1 -- 172.21.15.5:0/2881264421 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0809fd80 con 0x7f8f0813ad30 2024-09-13T11:42:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f07fff640 1 --2- 172.21.15.5:0/2881264421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 0x7f8f0813a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0d368640 1 --2- 172.21.15.5:0/2881264421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8f0813ad30 0x7f8f0813d120 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009d90 tx=0x7f8f0001c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87611bda5354f16c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 1 -- 172.21.15.5:0/2881264421 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f000b2070 con 0x7f8f0813ad30 2024-09-13T11:42:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 10 monclient: dump: 2024-09-13T11:42:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 10 monclient: _finish_auth 0 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 10 monclient: _check_auth_tickets 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 5 monclient: authenticate success, global_id 4689 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 1 -- 172.21.15.5:0/2881264421 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f000a58e0 con 0x7f8f0813ad30 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 1 -- 172.21.15.5:0/2881264421 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f000a5cb0 con 0x7f8f0813ad30 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 4 set_mon_vals no callback set 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f077fe640 10 monclient: dump: 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 10 monclient: shutdown 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2881264421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8f0813ad30 msgr2=0x7f8f0813d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 1 --2- 172.21.15.5:0/2881264421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8f0813ad30 0x7f8f0813d120 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009d90 tx=0x7f8f0001c800 comp rx=0 tx=0).stop 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2881264421 shutdown_connections 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 1 --2- 172.21.15.5:0/2881264421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8f0813ad30 0x7f8f0813d120 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 1 --2- 172.21.15.5:0/2881264421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 0x7f8f0813a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2881264421 >> 172.21.15.5:0/2881264421 conn(0x7f8f080a2930 msgr2=0x7f8f080a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2881264421 shutdown_connections 2024-09-13T11:42:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.859+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2881264421 wait complete. 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient: build_initial_monmap 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient: monmap: 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 1 Processor -- start 2024-09-13T11:42:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 1 -- start start 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient: init 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8f080a0ea0 0x7f8f0811e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient(hunting): picked mon.noname-a con 0x7f8f080a0ea0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 0x7f8f08135580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient(hunting): picked mon.noname-c con 0x7f8f080a1ae0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f0813ad30 0x7f8f081cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient(hunting): picked mon.noname-b con 0x7f8f0813ad30 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8f08140000 con 0x7f8f080a0ea0 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0813fe80 con 0x7f8f0813ad30 2024-09-13T11:42:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f8f08140180 con 0x7f8f080a1ae0 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient(hunting): authenticate will time out at 1138.213460s 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 0x7f8f08135580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8f080a0ea0 0x7f8f0811e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 0x7f8f08135580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:55174/0 (socket says 172.21.15.5:55174) 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 1 -- 172.21.15.5:0/2440335082 learned_addr learned my addr 172.21.15.5:0/2440335082 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 10 monclient(hunting): get_auth_request con 0x7f8f080a1ae0 auth_method 0 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 10 monclient(hunting): get_auth_request con 0x7f8f080a0ea0 auth_method 0 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 10 monclient(hunting): handle_auth_done global_id 4695 payload 824 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0d368640 1 --2- 172.21.15.5:0/2440335082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f0813ad30 0x7f8f081cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 1 -- 172.21.15.5:0/2440335082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 msgr2=0x7f8f08135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 1 --2- 172.21.15.5:0/2440335082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 0x7f8f08135580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 1 -- 172.21.15.5:0/2440335082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f0813ad30 msgr2=0x7f8f081cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 1 --2- 172.21.15.5:0/2440335082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f0813ad30 0x7f8f081cd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 10 monclient: _finish_hunting 0 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 1 monclient: found mon.noname-a 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 1 -- 172.21.15.5:0/2440335082 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0813b470 con 0x7f8f080a0ea0 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0d368640 10 monclient: get_auth_request con 0x7f8f0813ad30 auth_method 0 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 1 --2- 172.21.15.5:0/2440335082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 0x7f8f08135580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0d368640 1 --2- 172.21.15.5:0/2440335082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f0813ad30 0x7f8f081cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0cb67640 1 --2- 172.21.15.5:0/2440335082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8f080a0ea0 0x7f8f0811e710 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800c3c0 tx=0x7f8ef800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 1 -- 172.21.15.5:0/2440335082 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8017070 con 0x7f8f080a0ea0 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 10 monclient: dump: 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 10 monclient: _finish_auth 0 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 10 monclient: _check_auth_tickets 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 1 -- 172.21.15.5:0/2440335082 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ef8005d10 con 0x7f8f080a0ea0 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 5 monclient: authenticate success, global_id 4695 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient: _renew_subs 2024-09-13T11:42:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2440335082 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0806dbf0 con 0x7f8f080a0ea0 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 1 -- 172.21.15.5:0/2440335082 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8007310 con 0x7f8f080a0ea0 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 10 monclient: dump: 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:14.882 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f06ffd640 4 set_mon_vals no callback set 2024-09-13T11:42:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient: _renew_subs 2024-09-13T11:42:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2440335082 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f08136220 con 0x7f8f080a0ea0 2024-09-13T11:42:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2440335082 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f080a1280 con 0x7f8f080a0ea0 2024-09-13T11:42:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 1 -- 172.21.15.5:0/2440335082 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f8ef801a070 con 0x7f8f080a0ea0 2024-09-13T11:42:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 1 --2- 172.21.15.5:0/2440335082 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8ee803b2d0 0x7f8ee803d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f057fa640 1 -- 172.21.15.5:0/2440335082 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f8ef804dfe0 con 0x7f8f080a0ea0 2024-09-13T11:42:14.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 1 --2- 172.21.15.5:0/2440335082 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8ee803b2d0 0x7f8ee803d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:14.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.863+0000 7f8f07fff640 10 monclient: get_auth_request con 0x7f8ee803b2d0 auth_method 0 2024-09-13T11:42:14.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.867+0000 7f8f07fff640 1 --2- 172.21.15.5:0/2440335082 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8ee803b2d0 0x7f8ee803d790 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8f0813b980 tx=0x7f8ef0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:14.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.867+0000 7f8f057fa640 1 -- 172.21.15.5:0/2440335082 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f8ef804cec0 con 0x7f8f080a0ea0 2024-09-13T11:42:14.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.867+0000 7f8f057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:14.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:14.867+0000 7f8f057fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:15.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.027+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2440335082 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8f0811fdc0 con 0x7f8ee803b2d0 2024-09-13T11:42:15.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.031+0000 7f8f057fa640 1 -- 172.21.15.5:0/2440335082 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33921 (secure 0 0 0) 0x7f8ef0014440 con 0x7f8ee803b2d0 2024-09-13T11:42:15.036 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:42:15.036 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T11:42:15.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2440335082 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8ee803b2d0 msgr2=0x7f8ee803d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:15.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 1 --2- 172.21.15.5:0/2440335082 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8ee803b2d0 0x7f8ee803d790 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8f0813b980 tx=0x7f8ef0009190 comp rx=0 tx=0).stop 2024-09-13T11:42:15.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 10 monclient: shutdown 2024-09-13T11:42:15.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:15.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2440335082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8f080a0ea0 msgr2=0x7f8f0811e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:15.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 1 --2- 172.21.15.5:0/2440335082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8f080a0ea0 0x7f8f0811e710 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800c3c0 tx=0x7f8ef800c880 comp rx=0 tx=0).stop 2024-09-13T11:42:15.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2440335082 shutdown_connections 2024-09-13T11:42:15.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 1 --2- 172.21.15.5:0/2440335082 >> v2:172.21.15.39:6812/1340838781 conn(0x7f8ee803b2d0 0x7f8ee803d790 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:15.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 1 --2- 172.21.15.5:0/2440335082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f0813ad30 0x7f8f081cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:15.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 1 --2- 172.21.15.5:0/2440335082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8f080a1ae0 0x7f8f08135580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:15.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 1 --2- 172.21.15.5:0/2440335082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8f080a0ea0 0x7f8f0811e710 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:15.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2440335082 >> 172.21.15.5:0/2440335082 conn(0x7f8f080a2930 msgr2=0x7f8f0813cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:15.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2440335082 shutdown_connections 2024-09-13T11:42:15.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:15.035+0000 7f8f0edcb640 1 -- 172.21.15.5:0/2440335082 wait complete. 2024-09-13T11:42:15.054 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-13T11:42:14.330941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167812,"kb_used_data":1464,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063228,"statfs":{"total":579820584960,"available":579648745472,"internally_reserved":0,"allocated":1499136,"data_stored":1545562,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003328"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.252094+0000","last_change":"2024-09-13T11:40:14.135514+0000","last_active":"2024-09-13T11:40:41.252094+0000","last_peered":"2024-09-13T11:40:41.252094+0000","last_clean":"2024-09-13T11:40:41.252094+0000","last_became_active":"2024-09-13T11:40:12.461041+0000","last_became_peered":"2024-09-13T11:40:12.461041+0000","last_unstale":"2024-09-13T11:40:41.252094+0000","last_undegraded":"2024-09-13T11:40:41.252094+0000","last_fullsized":"2024-09-13T11:40:41.252094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:50.409094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00126878,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.466426+0000","last_change":"2024-09-13T11:40:14.143285+0000","last_active":"2024-09-13T11:40:41.466426+0000","last_peered":"2024-09-13T11:40:41.466426+0000","last_clean":"2024-09-13T11:40:41.466426+0000","last_became_active":"2024-09-13T11:40:12.106972+0000","last_became_peered":"2024-09-13T11:40:12.106972+0000","last_unstale":"2024-09-13T11:40:41.466426+0000","last_undegraded":"2024-09-13T11:40:41.466426+0000","last_fullsized":"2024-09-13T11:40:41.466426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:29.565401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001297091,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.252642+0000","last_change":"2024-09-13T11:40:14.135429+0000","last_active":"2024-09-13T11:40:41.252642+0000","last_peered":"2024-09-13T11:40:41.252642+0000","last_clean":"2024-09-13T11:40:41.252642+0000","last_became_active":"2024-09-13T11:40:12.120816+0000","last_became_peered":"2024-09-13T11:40:12.120816+0000","last_unstale":"2024-09-13T11:40:41.252642+0000","last_undegraded":"2024-09-13T11:40:41.252642+0000","last_fullsized":"2024-09-13T11:40:41.252642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:18:42.012635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011926969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.466384+0000","last_change":"2024-09-13T11:40:14.143208+0000","last_active":"2024-09-13T11:40:41.466384+0000","last_peered":"2024-09-13T11:40:41.466384+0000","last_clean":"2024-09-13T11:40:41.466384+0000","last_became_active":"2024-09-13T11:40:12.120092+0000","last_became_peered":"2024-09-13T11:40:12.120092+0000","last_unstale":"2024-09-13T11:40:41.466384+0000","last_undegraded":"2024-09-13T11:40:41.466384+0000","last_fullsized":"2024-09-13T11:40:41.466384+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:10.230543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001217161,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":36,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.461936+0000","last_change":"2024-09-13T11:40:14.139720+0000","last_active":"2024-09-13T11:40:41.461936+0000","last_peered":"2024-09-13T11:40:41.461936+0000","last_clean":"2024-09-13T11:40:41.461936+0000","last_became_active":"2024-09-13T11:40:12.110703+0000","last_became_peered":"2024-09-13T11:40:12.110703+0000","last_unstale":"2024-09-13T11:40:41.461936+0000","last_undegraded":"2024-09-13T11:40:41.461936+0000","last_fullsized":"2024-09-13T11:40:41.461936+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:50:57.779487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021711669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.250163+0000","last_change":"2024-09-13T11:40:14.131038+0000","last_active":"2024-09-13T11:40:41.250163+0000","last_peered":"2024-09-13T11:40:41.250163+0000","last_clean":"2024-09-13T11:40:41.250163+0000","last_became_active":"2024-09-13T11:40:12.102142+0000","last_became_peered":"2024-09-13T11:40:12.102142+0000","last_unstale":"2024-09-13T11:40:41.250163+0000","last_undegraded":"2024-09-13T11:40:41.250163+0000","last_fullsized":"2024-09-13T11:40:41.250163+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:15.679795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046429,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.251587+0000","last_change":"2024-09-13T11:40:14.135976+0000","last_active":"2024-09-13T11:40:41.251587+0000","last_peered":"2024-09-13T11:40:41.251587+0000","last_clean":"2024-09-13T11:40:41.251587+0000","last_became_active":"2024-09-13T11:40:12.105280+0000","last_became_peered":"2024-09-13T11:40:12.105280+0000","last_unstale":"2024-09-13T11:40:41.251587+0000","last_undegraded":"2024-09-13T11:40:41.251587+0000","last_fullsized":"2024-09-13T11:40:41.251587+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:20.791570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001476088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.462629+0000","last_change":"2024-09-13T11:40:14.138397+0000","last_active":"2024-09-13T11:40:41.462629+0000","last_peered":"2024-09-13T11:40:41.462629+0000","last_clean":"2024-09-13T11:40:41.462629+0000","last_became_active":"2024-09-13T11:40:12.113752+0000","last_became_peered":"2024-09-13T11:40:12.113752+0000","last_unstale":"2024-09-13T11:40:41.462629+0000","last_undegraded":"2024-09-13T11:40:41.462629+0000","last_fullsized":"2024-09-13T11:40:41.462629+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:11.082334+0000","last_clean_scrub_stamp":"2024-09-13T11:40:11.082334+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:50:52.508777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086960399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":126,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:40:41.252004+0000","last_change":"2024-09-13T11:40:05.394873+0000","last_active":"2024-09-13T11:40:41.252004+0000","last_peered":"2024-09-13T11:40:41.252004+0000","last_clean":"2024-09-13T11:40:41.252004+0000","last_became_active":"2024-09-13T11:40:05.394176+0000","last_became_peered":"2024-09-13T11:40:05.394176+0000","last_unstale":"2024-09-13T11:40:41.252004+0000","last_undegraded":"2024-09-13T11:40:41.252004+0000","last_fullsized":"2024-09-13T11:40:41.252004+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:40:04.377013+0000","last_clean_scrub_stamp":"2024-09-13T11:40:04.377013+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:00:31.498589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607580,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":81297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607580,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":229376,"data_stored":81278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607580,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":286720,"data_stored":610206,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607580,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":81297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640285,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":229376,"data_stored":81278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640285,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":610206,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:42:15.058 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T11:40:11.082334+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=40, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=42, tm_sec=10, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:42:15.058 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-13T11:40:11.082334+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=40, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=42, tm_sec=10, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:42:15.058 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-13T11:40:11.082334+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=40, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=42, tm_sec=10, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:42:15.058 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-13T11:40:11.082334+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=40, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=42, tm_sec=10, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:42:15.059 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-13T11:40:11.082334+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=40, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=42, tm_sec=10, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:42:15.059 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-13T11:40:11.082334+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=40, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=42, tm_sec=10, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:42:15.059 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-13T11:40:11.082334+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=40, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=42, tm_sec=10, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:42:15.059 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T11:40:11.082334+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=40, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=42, tm_sec=10, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:42:15.059 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T11:40:04.377013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=40, tm_sec=4, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=42, tm_sec=10, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:42:15.059 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T11:42:35.061 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:42:35.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.190+0000 7f6f99001640 10 monclient: get_monmap_and_config 2024-09-13T11:42:35.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.190+0000 7f6f99001640 10 monclient: build_initial_monmap 2024-09-13T11:42:35.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.190+0000 7f6f99001640 10 monclient: monmap: 2024-09-13T11:42:35.200 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:35.200 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:35.200 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:35.195903+0000 2024-09-13T11:42:35.200 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:35.195903+0000 2024-09-13T11:42:35.200 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:35.200 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:35.200 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:35.200 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:35.200 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:35.200 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:35.200 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.190+0000 7f6f99001640 1 Processor -- start 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 1 -- start start 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 10 monclient: init 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6f94134810 0x7f6f94134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 10 monclient(hunting): picked mon.noname-c con 0x7f6f94134810 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6f9409d1d0 0x7f6f9409d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 10 monclient(hunting): picked mon.noname-a con 0x7f6f9409d1d0 addr v2:172.21.15.5:3300/0 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409db20 0x7f6f9409dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 10 monclient(hunting): picked mon.noname-b con 0x7f6f9409db20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f6f940a0950 con 0x7f6f9409d1d0 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7f6f9409ce60 con 0x7f6f94134810 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f6f9409eeb0 con 0x7f6f9409db20 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f99001640 10 monclient(hunting): authenticate will time out at 1158.544771s 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409db20 0x7f6f9409dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409db20 0x7f6f9409dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:36798/0 (socket says 172.21.15.5:36798) 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 1 -- 172.21.15.5:0/1774069370 learned_addr learned my addr 172.21.15.5:0/1774069370 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 10 monclient(hunting): get_auth_request con 0x7f6f9409db20 auth_method 0 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9259c640 1 --2- 172.21.15.5:0/1774069370 >> v2:172.21.15.5:3300/0 conn(0x7f6f9409d1d0 0x7f6f9409d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9259c640 10 monclient(hunting): get_auth_request con 0x7f6f9409d1d0 auth_method 0 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9259c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f92d9d640 1 --2- 172.21.15.5:0/1774069370 >> v2:172.21.15.39:3300/0 conn(0x7f6f94134810 0x7f6f94134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f92d9d640 10 monclient(hunting): get_auth_request con 0x7f6f94134810 auth_method 0 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 10 monclient(hunting): handle_auth_done global_id 4517 payload 274 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 1 -- 172.21.15.5:0/1774069370 >> v2:172.21.15.39:3300/0 conn(0x7f6f94134810 msgr2=0x7f6f94134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 1 --2- 172.21.15.5:0/1774069370 >> v2:172.21.15.39:3300/0 conn(0x7f6f94134810 0x7f6f94134bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 1 -- 172.21.15.5:0/1774069370 >> v2:172.21.15.5:3300/0 conn(0x7f6f9409d1d0 msgr2=0x7f6f9409d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 1 --2- 172.21.15.5:0/1774069370 >> v2:172.21.15.5:3300/0 conn(0x7f6f9409d1d0 0x7f6f9409d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 10 monclient: _finish_hunting 0 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 1 monclient: found mon.noname-b 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:42:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 1 -- 172.21.15.5:0/1774069370 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f94135130 con 0x7f6f9409db20 2024-09-13T11:42:35.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9259c640 1 --2- 172.21.15.5:0/1774069370 >> v2:172.21.15.5:3300/0 conn(0x7f6f9409d1d0 0x7f6f9409d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T11:42:35.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.194+0000 7f6f9359e640 1 --2- 172.21.15.5:0/1774069370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409db20 0x7f6f9409dfb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6f940a2630 tx=0x7f6f8401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30793ebe0e2811f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 1 -- 172.21.15.5:0/1774069370 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f840b0070 con 0x7f6f9409db20 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 10 monclient: dump: 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 10 monclient: _finish_auth 0 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 10 monclient: _check_auth_tickets 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 5 monclient: authenticate success, global_id 4517 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 1 -- 172.21.15.5:0/1774069370 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f840ab600 con 0x7f6f9409db20 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 1 -- 172.21.15.5:0/1774069370 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f840ab9d0 con 0x7f6f9409db20 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 4 set_mon_vals no callback set 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:35.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f91d9b640 10 monclient: dump: 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient: shutdown 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 -- 172.21.15.5:0/1774069370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409db20 msgr2=0x7f6f9409dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 --2- 172.21.15.5:0/1774069370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409db20 0x7f6f9409dfb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6f940a2630 tx=0x7f6f8401c3e0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 -- 172.21.15.5:0/1774069370 shutdown_connections 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 --2- 172.21.15.5:0/1774069370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409db20 0x7f6f9409dfb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 --2- 172.21.15.5:0/1774069370 >> v2:172.21.15.5:3300/0 conn(0x7f6f9409d1d0 0x7f6f9409d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 --2- 172.21.15.5:0/1774069370 >> v2:172.21.15.39:3300/0 conn(0x7f6f94134810 0x7f6f94134bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 -- 172.21.15.5:0/1774069370 >> 172.21.15.5:0/1774069370 conn(0x7f6f940a2930 msgr2=0x7f6f940a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 -- 172.21.15.5:0/1774069370 shutdown_connections 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 -- 172.21.15.5:0/1774069370 wait complete. 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient: build_initial_monmap 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient: monmap: 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:35.215 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 Processor -- start 2024-09-13T11:42:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 -- start start 2024-09-13T11:42:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient: init 2024-09-13T11:42:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-13T11:42:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409d1d0 0x7f6f94139dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient(hunting): picked mon.noname-c con 0x7f6f9409d1d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f9409db20 0x7f6f9413a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient(hunting): picked mon.noname-a con 0x7f6f9409db20 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f94134810 0x7f6f9413a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient(hunting): picked mon.noname-b con 0x7f6f94134810 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f6f940a16c0 con 0x7f6f9409db20 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6f940a1540 con 0x7f6f94134810 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7f6f940a1840 con 0x7f6f9409d1d0 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient(hunting): authenticate will time out at 1158.550464s 2024-09-13T11:42:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f92d9d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409d1d0 0x7f6f94139dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f9409db20 0x7f6f9413a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f92d9d640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409d1d0 0x7f6f94139dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:36808/0 (socket says 172.21.15.5:36808) 2024-09-13T11:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f92d9d640 1 -- 172.21.15.5:0/771239540 learned_addr learned my addr 172.21.15.5:0/771239540 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f92d9d640 10 monclient(hunting): get_auth_request con 0x7f6f9409d1d0 auth_method 0 2024-09-13T11:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f9409db20 0x7f6f9413a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:40184/0 (socket says 172.21.15.5:40184) 2024-09-13T11:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 10 monclient(hunting): get_auth_request con 0x7f6f9409db20 auth_method 0 2024-09-13T11:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9359e640 1 --2- 172.21.15.5:0/771239540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f94134810 0x7f6f9413a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9359e640 10 monclient(hunting): get_auth_request con 0x7f6f94134810 auth_method 0 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9359e640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 10 monclient(hunting): handle_auth_done global_id 4701 payload 824 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 1 -- 172.21.15.5:0/771239540 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409d1d0 msgr2=0x7f6f94139dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 1 --2- 172.21.15.5:0/771239540 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409d1d0 0x7f6f94139dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 1 -- 172.21.15.5:0/771239540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f94134810 msgr2=0x7f6f9413a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 1 --2- 172.21.15.5:0/771239540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f94134810 0x7f6f9413a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 10 monclient: _finish_hunting 0 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 1 monclient: found mon.noname-a 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 1 -- 172.21.15.5:0/771239540 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9411e810 con 0x7f6f9409db20 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9259c640 1 --2- 172.21.15.5:0/771239540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f9409db20 0x7f6f9413a300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c00e9b0 tx=0x7f6f7c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 1 -- 172.21.15.5:0/771239540 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c017070 con 0x7f6f9409db20 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 10 monclient: dump: 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 10 monclient: _finish_auth 0 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 10 monclient: _check_auth_tickets 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 5 monclient: authenticate success, global_id 4701 2024-09-13T11:42:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 1 -- 172.21.15.5:0/771239540 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f7c00c9a0 con 0x7f6f9409db20 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient: _renew_subs 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 -- 172.21.15.5:0/771239540 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f9409c420 con 0x7f6f9409db20 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 1 -- 172.21.15.5:0/771239540 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c00cd90 con 0x7f6f9409db20 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f8b7fe640 10 monclient: dump: 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient: _renew_subs 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f9159a640 4 set_mon_vals no callback set 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.198+0000 7f6f99001640 1 -- 172.21.15.5:0/771239540 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f9411c4b0 con 0x7f6f9409db20 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.202+0000 7f6f99001640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.202+0000 7f6f99001640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.202+0000 7f6f99001640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.202+0000 7f6f99001640 1 -- 172.21.15.5:0/771239540 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f940a1c70 con 0x7f6f9409db20 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.202+0000 7f6f8b7fe640 1 -- 172.21.15.5:0/771239540 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7f6f7c01a070 con 0x7f6f9409db20 2024-09-13T11:42:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.202+0000 7f6f8b7fe640 1 --2- 172.21.15.5:0/771239540 >> v2:172.21.15.39:6812/1340838781 conn(0x7f6f7403b2d0 0x7f6f7403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.202+0000 7f6f8b7fe640 1 -- 172.21.15.5:0/771239540 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7f6f7c04ead0 con 0x7f6f9409db20 2024-09-13T11:42:35.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.202+0000 7f6f92d9d640 1 --2- 172.21.15.5:0/771239540 >> v2:172.21.15.39:6812/1340838781 conn(0x7f6f7403b2d0 0x7f6f7403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.202+0000 7f6f92d9d640 10 monclient: get_auth_request con 0x7f6f7403b2d0 auth_method 0 2024-09-13T11:42:35.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.202+0000 7f6f92d9d640 1 --2- 172.21.15.5:0/771239540 >> v2:172.21.15.39:6812/1340838781 conn(0x7f6f7403b2d0 0x7f6f7403d790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c003b90 tx=0x7f6f8c009970 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:35.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.206+0000 7f6f8b7fe640 1 -- 172.21.15.5:0/771239540 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f6f7c011350 con 0x7f6f9409db20 2024-09-13T11:42:35.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.206+0000 7f6f8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:35.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.206+0000 7f6f8b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:35.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.374+0000 7f6f99001640 1 -- 172.21.15.5:0/771239540 --> v2:172.21.15.39:6812/1340838781 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6f94069ff0 con 0x7f6f7403b2d0 2024-09-13T11:42:35.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.374+0000 7f6f8b7fe640 1 -- 172.21.15.5:0/771239540 <== mgr.4105 v2:172.21.15.39:6812/1340838781 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33929 (secure 0 0 0) 0x7f6f94069ff0 con 0x7f6f7403b2d0 2024-09-13T11:42:35.381 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:42:35.381 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T11:42:35.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 1 -- 172.21.15.5:0/771239540 >> v2:172.21.15.39:6812/1340838781 conn(0x7f6f7403b2d0 msgr2=0x7f6f7403d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:35.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 1 --2- 172.21.15.5:0/771239540 >> v2:172.21.15.39:6812/1340838781 conn(0x7f6f7403b2d0 0x7f6f7403d790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c003b90 tx=0x7f6f8c009970 comp rx=0 tx=0).stop 2024-09-13T11:42:35.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 10 monclient: shutdown 2024-09-13T11:42:35.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:35.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 1 -- 172.21.15.5:0/771239540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f9409db20 msgr2=0x7f6f9413a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:35.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 1 --2- 172.21.15.5:0/771239540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f9409db20 0x7f6f9413a300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c00e9b0 tx=0x7f6f7c00ee70 comp rx=0 tx=0).stop 2024-09-13T11:42:35.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 1 -- 172.21.15.5:0/771239540 shutdown_connections 2024-09-13T11:42:35.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 1 --2- 172.21.15.5:0/771239540 >> v2:172.21.15.39:6812/1340838781 conn(0x7f6f7403b2d0 0x7f6f7403d790 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 1 --2- 172.21.15.5:0/771239540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f94134810 0x7f6f9413a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 1 --2- 172.21.15.5:0/771239540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f9409db20 0x7f6f9413a300 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 1 --2- 172.21.15.5:0/771239540 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6f9409d1d0 0x7f6f94139dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 1 -- 172.21.15.5:0/771239540 >> 172.21.15.5:0/771239540 conn(0x7f6f940a2930 msgr2=0x7f6f94071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:35.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 1 -- 172.21.15.5:0/771239540 shutdown_connections 2024-09-13T11:42:35.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.378+0000 7f6f99001640 1 -- 172.21.15.5:0/771239540 wait complete. 2024-09-13T11:42:35.400 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-09-13T11:42:34.336179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167776,"kb_used_data":1464,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063264,"statfs":{"total":579820584960,"available":579648782336,"internally_reserved":0,"allocated":1499136,"data_stored":1545562,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003101"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:42:16.772934+0000","last_change":"2024-09-13T11:42:16.772762+0000","last_active":"2024-09-13T11:42:16.772934+0000","last_peered":"2024-09-13T11:42:16.772934+0000","last_clean":"2024-09-13T11:42:16.772934+0000","last_became_active":"2024-09-13T11:40:12.461041+0000","last_became_peered":"2024-09-13T11:40:12.461041+0000","last_unstale":"2024-09-13T11:42:16.772934+0000","last_undegraded":"2024-09-13T11:42:16.772934+0000","last_fullsized":"2024-09-13T11:42:16.772934+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:42:16.772598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:42:16.772598+0000","last_clean_scrub_stamp":"2024-09-13T11:42:16.772598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:48:49.890807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00126878,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:42:12.895868+0000","last_change":"2024-09-13T11:42:12.895764+0000","last_active":"2024-09-13T11:42:12.895868+0000","last_peered":"2024-09-13T11:42:12.895868+0000","last_clean":"2024-09-13T11:42:12.895868+0000","last_became_active":"2024-09-13T11:40:12.106972+0000","last_became_peered":"2024-09-13T11:40:12.106972+0000","last_unstale":"2024-09-13T11:42:12.895868+0000","last_undegraded":"2024-09-13T11:42:12.895868+0000","last_fullsized":"2024-09-13T11:42:12.895868+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:42:12.895672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:42:12.895672+0000","last_clean_scrub_stamp":"2024-09-13T11:42:12.895672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:15.285287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001297091,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:42:15.756647+0000","last_change":"2024-09-13T11:42:15.756486+0000","last_active":"2024-09-13T11:42:15.756647+0000","last_peered":"2024-09-13T11:42:15.756647+0000","last_clean":"2024-09-13T11:42:15.756647+0000","last_became_active":"2024-09-13T11:40:12.120816+0000","last_became_peered":"2024-09-13T11:40:12.120816+0000","last_unstale":"2024-09-13T11:42:15.756647+0000","last_undegraded":"2024-09-13T11:42:15.756647+0000","last_fullsized":"2024-09-13T11:42:15.756647+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:42:15.756286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:42:15.756286+0000","last_clean_scrub_stamp":"2024-09-13T11:42:15.756286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:36.528266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011926969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:42:13.904618+0000","last_change":"2024-09-13T11:42:13.904561+0000","last_active":"2024-09-13T11:42:13.904618+0000","last_peered":"2024-09-13T11:42:13.904618+0000","last_clean":"2024-09-13T11:42:13.904618+0000","last_became_active":"2024-09-13T11:40:12.120092+0000","last_became_peered":"2024-09-13T11:40:12.120092+0000","last_unstale":"2024-09-13T11:42:13.904618+0000","last_undegraded":"2024-09-13T11:42:13.904618+0000","last_fullsized":"2024-09-13T11:42:13.904618+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:42:13.904495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:42:13.904495+0000","last_clean_scrub_stamp":"2024-09-13T11:42:13.904495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:45:39.901423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001217161,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"26'3","reported_seq":46,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:42:15.250715+0000","last_change":"2024-09-13T11:42:15.250578+0000","last_active":"2024-09-13T11:42:15.250715+0000","last_peered":"2024-09-13T11:42:15.250715+0000","last_clean":"2024-09-13T11:42:15.250715+0000","last_became_active":"2024-09-13T11:40:12.110703+0000","last_became_peered":"2024-09-13T11:40:12.110703+0000","last_unstale":"2024-09-13T11:42:15.250715+0000","last_undegraded":"2024-09-13T11:42:15.250715+0000","last_fullsized":"2024-09-13T11:42:15.250715+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'3","last_scrub_stamp":"2024-09-13T11:42:15.250397+0000","last_deep_scrub":"26'3","last_deep_scrub_stamp":"2024-09-13T11:42:15.250397+0000","last_clean_scrub_stamp":"2024-09-13T11:42:15.250397+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:07.317366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021711669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:42:13.942779+0000","last_change":"2024-09-13T11:42:13.942732+0000","last_active":"2024-09-13T11:42:13.942779+0000","last_peered":"2024-09-13T11:42:13.942779+0000","last_clean":"2024-09-13T11:42:13.942779+0000","last_became_active":"2024-09-13T11:40:12.102142+0000","last_became_peered":"2024-09-13T11:40:12.102142+0000","last_unstale":"2024-09-13T11:42:13.942779+0000","last_undegraded":"2024-09-13T11:42:13.942779+0000","last_fullsized":"2024-09-13T11:42:13.942779+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:42:13.942652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:42:13.942652+0000","last_clean_scrub_stamp":"2024-09-13T11:42:13.942652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:06:55.982750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046429,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:42:14.777599+0000","last_change":"2024-09-13T11:42:14.777599+0000","last_active":"2024-09-13T11:42:14.777599+0000","last_peered":"2024-09-13T11:42:14.777599+0000","last_clean":"2024-09-13T11:42:14.777599+0000","last_became_active":"2024-09-13T11:40:12.105280+0000","last_became_peered":"2024-09-13T11:40:12.105280+0000","last_unstale":"2024-09-13T11:42:14.777599+0000","last_undegraded":"2024-09-13T11:42:14.777599+0000","last_fullsized":"2024-09-13T11:42:14.777599+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:42:14.777394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:42:14.777394+0000","last_clean_scrub_stamp":"2024-09-13T11:42:14.777394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:45:00.372759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001476088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"26'2","reported_seq":45,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:42:16.230247+0000","last_change":"2024-09-13T11:42:16.230109+0000","last_active":"2024-09-13T11:42:16.230247+0000","last_peered":"2024-09-13T11:42:16.230247+0000","last_clean":"2024-09-13T11:42:16.230247+0000","last_became_active":"2024-09-13T11:40:12.113752+0000","last_became_peered":"2024-09-13T11:40:12.113752+0000","last_unstale":"2024-09-13T11:42:16.230247+0000","last_undegraded":"2024-09-13T11:42:16.230247+0000","last_fullsized":"2024-09-13T11:42:16.230247+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'2","last_scrub_stamp":"2024-09-13T11:42:16.229941+0000","last_deep_scrub":"26'2","last_deep_scrub_stamp":"2024-09-13T11:42:16.229941+0000","last_clean_scrub_stamp":"2024-09-13T11:42:16.229941+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:08.623831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00086960399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"26'74","reported_seq":136,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-13T11:42:13.769542+0000","last_change":"2024-09-13T11:42:13.769542+0000","last_active":"2024-09-13T11:42:13.769542+0000","last_peered":"2024-09-13T11:42:13.769542+0000","last_clean":"2024-09-13T11:42:13.769542+0000","last_became_active":"2024-09-13T11:40:05.394176+0000","last_became_peered":"2024-09-13T11:40:05.394176+0000","last_unstale":"2024-09-13T11:42:13.769542+0000","last_undegraded":"2024-09-13T11:42:13.769542+0000","last_fullsized":"2024-09-13T11:42:13.769542+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'74","last_scrub_stamp":"2024-09-13T11:42:13.769368+0000","last_deep_scrub":"26'74","last_deep_scrub_stamp":"2024-09-13T11:42:13.769368+0000","last_clean_scrub_stamp":"2024-09-13T11:42:13.769368+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:07.727783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":19462,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607584,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":81297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":229376,"data_stored":81278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607584,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":610206,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607584,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":81297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640289,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":229376,"data_stored":81278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640289,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":610206,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":9731,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:42:35.401 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T11:42:35.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient: get_monmap_and_config 2024-09-13T11:42:35.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient: build_initial_monmap 2024-09-13T11:42:35.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient: monmap: 2024-09-13T11:42:35.527 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:35.527 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:35.527 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:42:35.526711+0000 2024-09-13T11:42:35.527 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:42:35.526711+0000 2024-09-13T11:42:35.528 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:35.528 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:35.528 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-13T11:42:35.528 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.39:6789/0 mon.noname-c-legacy 2024-09-13T11:42:35.528 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-13T11:42:35.528 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-13T11:42:35.528 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.39:3300/0 mon.noname-c 2024-09-13T11:42:35.528 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:35.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 1 Processor -- start 2024-09-13T11:42:35.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 1 -- start start 2024-09-13T11:42:35.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient: init 2024-09-13T11:42:35.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:35.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-13T11:42:35.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae44134a70 addr v1:172.21.15.39:6789/0 2024-09-13T11:42:35.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae4409ee00 0x7fae441397f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient(hunting): picked mon.noname-b con 0x7fae4409ee00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:35.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fae44139d30 0x7fae4413c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient(hunting): picked mon.noname-c con 0x7fae44139d30 addr v2:172.21.15.39:3300/0 2024-09-13T11:42:35.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae4409e420 con 0x7fae44134a70 2024-09-13T11:42:35.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:35.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 -- 0x7fae4409e140 con 0x7fae44139d30 2024-09-13T11:42:35.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:35.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fae4409e700 con 0x7fae4409ee00 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.522+0000 7fae498b7640 10 monclient(hunting): authenticate will time out at 1158.875957s 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae4409ee00 0x7fae441397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae4409ee00 0x7fae441397f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:36818/0 (socket says 172.21.15.5:36818) 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 1 -- 172.21.15.5:0/2908508847 learned_addr learned my addr 172.21.15.5:0/2908508847 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 10 monclient(hunting): get_auth_request con 0x7fae4409ee00 auth_method 0 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae437fe640 1 --2- 172.21.15.5:0/2908508847 >> v2:172.21.15.39:3300/0 conn(0x7fae44139d30 0x7fae4413c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:35.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae437fe640 10 monclient(hunting): get_auth_request con 0x7fae44139d30 auth_method 0 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae437fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 10 monclient(hunting): handle_auth_done global_id 4523 payload 274 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 1 -- 172.21.15.5:0/2908508847 >> v2:172.21.15.39:3300/0 conn(0x7fae44139d30 msgr2=0x7fae4413c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 1 --2- 172.21.15.5:0/2908508847 >> v2:172.21.15.39:3300/0 conn(0x7fae44139d30 0x7fae4413c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 1 -- 172.21.15.5:0/2908508847 >> v1:172.21.15.39:6789/0 conn(0x7fae44134a70 legacy=0x7fae44134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 10 monclient: _finish_hunting 0 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 1 monclient: found mon.noname-b 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 1 -- 172.21.15.5:0/2908508847 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae4409d8f0 con 0x7fae4409ee00 2024-09-13T11:42:35.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae437fe640 1 --2- 172.21.15.5:0/2908508847 >> v2:172.21.15.39:3300/0 conn(0x7fae44139d30 0x7fae4413c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:42:35.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae3bfff640 1 --2- 172.21.15.5:0/2908508847 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae4409ee00 0x7fae441397f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00bb10 tx=0x7fae2c01eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96e4ba912998c8e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:35.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 1 -- 172.21.15.5:0/2908508847 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0b4070 con 0x7fae4409ee00 2024-09-13T11:42:35.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:35.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T11:42:35.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 10 monclient: dump: 2024-09-13T11:42:35.538 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:35.538 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 10 monclient: _finish_auth 0 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 10 monclient: _check_auth_tickets 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 1 -- 172.21.15.5:0/2908508847 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae2c0a7950 con 0x7fae4409ee00 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 5 monclient: authenticate success, global_id 4523 2024-09-13T11:42:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 10 monclient: get_monmap_and_config success 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 1 -- 172.21.15.5:0/2908508847 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0a7d20 con 0x7fae4409ee00 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 4 set_mon_vals no callback set 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae427fc640 10 monclient: dump: 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 10 monclient: shutdown 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 1 -- 172.21.15.5:0/2908508847 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae4409ee00 msgr2=0x7fae441397f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 1 --2- 172.21.15.5:0/2908508847 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae4409ee00 0x7fae441397f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00bb10 tx=0x7fae2c01eb80 comp rx=0 tx=0).stop 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 1 -- 172.21.15.5:0/2908508847 shutdown_connections 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 1 --2- 172.21.15.5:0/2908508847 >> v2:172.21.15.39:3300/0 conn(0x7fae44139d30 0x7fae4413c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 1 --2- 172.21.15.5:0/2908508847 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae4409ee00 0x7fae441397f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 1 -- 172.21.15.5:0/2908508847 >> 172.21.15.5:0/2908508847 conn(0x7fae440a61e0 msgr2=0x7fae440a65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 1 -- 172.21.15.5:0/2908508847 shutdown_connections 2024-09-13T11:42:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.526+0000 7fae498b7640 1 -- 172.21.15.5:0/2908508847 wait complete. 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient: build_initial_monmap 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient: monmap: 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.noname-b 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 1 Processor -- start 2024-09-13T11:42:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 1 -- start start 2024-09-13T11:42:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient: init 2024-09-13T11:42:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient: _reopen_session rank -1 2024-09-13T11:42:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-13T11:42:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae4409ee00 0x7fae441cb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient(hunting): picked mon.noname-b con 0x7fae4409ee00 addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2024-09-13T11:42:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae44134a70 0x7fae441329c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient(hunting): picked mon.noname-a con 0x7fae44134a70 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T11:42:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae44139d30 0x7fae440717d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient(hunting): picked mon.noname-c con 0x7fae44139d30 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-13T11:42:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fae440a1380 con 0x7fae44134a70 2024-09-13T11:42:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fae440a1200 con 0x7fae4409ee00 2024-09-13T11:42:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient(hunting): start opening mon connection 2024-09-13T11:42:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 -- 0x7fae440a1500 con 0x7fae44139d30 2024-09-13T11:42:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient(hunting): _renew_subs 2024-09-13T11:42:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient(hunting): authenticate will time out at 1158.881397s 2024-09-13T11:42:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae437fe640 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae44139d30 0x7fae440717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae44134a70 0x7fae441329c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae44134a70 0x7fae441329c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:40198/0 (socket says 172.21.15.5:40198) 2024-09-13T11:42:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 1 -- 172.21.15.5:0/793416435 learned_addr learned my addr 172.21.15.5:0/793416435 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-13T11:42:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 10 monclient(hunting): get_auth_request con 0x7fae44134a70 auth_method 0 2024-09-13T11:42:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae437fe640 10 monclient(hunting): get_auth_request con 0x7fae44139d30 auth_method 0 2024-09-13T11:42:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae437fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae42ffd640 1 --2- 172.21.15.5:0/793416435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae4409ee00 0x7fae441cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-13T11:42:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-13T11:42:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-13T11:42:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae42ffd640 10 monclient(hunting): get_auth_request con 0x7fae4409ee00 auth_method 0 2024-09-13T11:42:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-13T11:42:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T11:42:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T11:42:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 10 monclient(hunting): handle_auth_done global_id 4704 payload 824 2024-09-13T11:42:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 1 -- 172.21.15.5:0/793416435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae44139d30 msgr2=0x7fae440717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 1 --2- 172.21.15.5:0/793416435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae44139d30 0x7fae440717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 1 -- 172.21.15.5:0/793416435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae4409ee00 msgr2=0x7fae441cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 1 --2- 172.21.15.5:0/793416435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae4409ee00 0x7fae441cb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 10 monclient: _finish_hunting 0 2024-09-13T11:42:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 1 monclient: found mon.noname-a 2024-09-13T11:42:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T11:42:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-13T11:42:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 1 -- 172.21.15.5:0/793416435 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae44074230 con 0x7fae44134a70 2024-09-13T11:42:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae42ffd640 1 --2- 172.21.15.5:0/793416435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae4409ee00 0x7fae441cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T11:42:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae3bfff640 1 --2- 172.21.15.5:0/793416435 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae44134a70 0x7fae441329c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fae2c01d230 tx=0x7fae2c01b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 1 -- 172.21.15.5:0/793416435 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0b4070 con 0x7fae44134a70 2024-09-13T11:42:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T11:42:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 10 monclient: dump: 2024-09-13T11:42:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 10 monclient: _finish_auth 0 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 10 monclient: _check_auth_tickets 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 1 -- 172.21.15.5:0/793416435 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae2c0b6070 con 0x7fae44134a70 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 5 monclient: authenticate success, global_id 4704 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient: _renew_subs 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 1 -- 172.21.15.5:0/793416435 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae4411d350 con 0x7fae44134a70 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 1 -- 172.21.15.5:0/793416435 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0afbf0 con 0x7fae44134a70 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T11:42:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 10 monclient: dump: 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 59046471-79a9-4d90-9616-da0b446053a7 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-13T11:39:20.750174+0000 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.b 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae41ffb640 4 set_mon_vals no callback set 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient: _renew_subs 2024-09-13T11:42:35.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 1 -- 172.21.15.5:0/793416435 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae44073210 con 0x7fae44134a70 2024-09-13T11:42:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae498b7640 1 -- 172.21.15.5:0/793416435 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae44134e50 con 0x7fae44134a70 2024-09-13T11:42:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 1 -- 172.21.15.5:0/793416435 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 48297+0+0 (secure 0 0 0) 0x7fae2c0c43f0 con 0x7fae44134a70 2024-09-13T11:42:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 1 --2- 172.21.15.5:0/793416435 >> v2:172.21.15.39:6812/1340838781 conn(0x7fae2403b2a0 0x7fae2403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T11:42:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae40ff9640 1 -- 172.21.15.5:0/793416435 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3506+0+0 (secure 0 0 0) 0x7fae2c0e9040 con 0x7fae44134a70 2024-09-13T11:42:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.530+0000 7fae42ffd640 1 --2- 172.21.15.5:0/793416435 >> v2:172.21.15.39:6812/1340838781 conn(0x7fae2403b2a0 0x7fae2403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T11:42:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.534+0000 7fae42ffd640 10 monclient: get_auth_request con 0x7fae2403b2a0 auth_method 0 2024-09-13T11:42:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.534+0000 7fae42ffd640 1 --2- 172.21.15.5:0/793416435 >> v2:172.21.15.39:6812/1340838781 conn(0x7fae2403b2a0 0x7fae2403d760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fae34000f70 tx=0x7fae34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T11:42:35.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.538+0000 7fae40ff9640 1 -- 172.21.15.5:0/793416435 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fae44074230 con 0x7fae44134a70 2024-09-13T11:42:35.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.538+0000 7fae40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T11:42:35.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.538+0000 7fae40ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T11:42:35.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.706+0000 7fae498b7640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-13T11:42:35.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.706+0000 7fae498b7640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-13T11:42:35.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.706+0000 7fae498b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-13T11:42:35.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.706+0000 7fae498b7640 1 -- 172.21.15.5:0/793416435 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fae4411ccf0 con 0x7fae44134a70 2024-09-13T11:42:35.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.710+0000 7fae40ff9640 1 -- 172.21.15.5:0/793416435 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v3)=0 v3) ==== 123+0+0 (secure 0 0 0) 0x7fae2c018800 con 0x7fae44134a70 2024-09-13T11:42:35.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.710+0000 7fae40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-13T11:42:35.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.710+0000 7fae40ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T11:42:35.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.710+0000 7fae40ff9640 1 -- 172.21.15.5:0/793416435 <== mon.0 v2:172.21.15.5:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fae2c0ae740 con 0x7fae44134a70 2024-09-13T11:42:35.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.710+0000 7fae40ff9640 10 monclient: handle_config config(1 keys) 2024-09-13T11:42:35.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.710+0000 7fae417fa640 4 set_mon_vals no callback set 2024-09-13T11:42:35.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.710+0000 7fae417fa640 10 set_mon_vals mon_health_to_clog = false 2024-09-13T11:42:35.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 1 -- 172.21.15.5:0/793416435 >> v2:172.21.15.39:6812/1340838781 conn(0x7fae2403b2a0 msgr2=0x7fae2403d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:35.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 1 --2- 172.21.15.5:0/793416435 >> v2:172.21.15.39:6812/1340838781 conn(0x7fae2403b2a0 0x7fae2403d760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fae34000f70 tx=0x7fae34009040 comp rx=0 tx=0).stop 2024-09-13T11:42:35.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 10 monclient: shutdown 2024-09-13T11:42:35.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T11:42:35.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 1 -- 172.21.15.5:0/793416435 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae44134a70 msgr2=0x7fae441329c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T11:42:35.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 1 --2- 172.21.15.5:0/793416435 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae44134a70 0x7fae441329c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fae2c01d230 tx=0x7fae2c01b920 comp rx=0 tx=0).stop 2024-09-13T11:42:35.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 1 -- 172.21.15.5:0/793416435 shutdown_connections 2024-09-13T11:42:35.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 1 --2- 172.21.15.5:0/793416435 >> v2:172.21.15.39:6812/1340838781 conn(0x7fae2403b2a0 0x7fae2403d760 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 1 --2- 172.21.15.5:0/793416435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fae44139d30 0x7fae440717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 1 --2- 172.21.15.5:0/793416435 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae44134a70 0x7fae441329c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 1 --2- 172.21.15.5:0/793416435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae4409ee00 0x7fae441cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T11:42:35.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 1 -- 172.21.15.5:0/793416435 >> 172.21.15.5:0/793416435 conn(0x7fae440a61e0 msgr2=0x7fae440a3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T11:42:35.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 1 -- 172.21.15.5:0/793416435 shutdown_connections 2024-09-13T11:42:35.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T11:42:35.714+0000 7fae498b7640 1 -- 172.21.15.5:0/793416435 wait complete. 2024-09-13T11:42:35.736 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T11:42:35.736 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T11:42:35.736 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T11:42:35.736 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:42:35.800 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T11:42:35.800 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T11:42:35.800 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:42:35.883 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T11:42:35.883 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T11:42:35.883 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:42:35.959 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T11:42:35.959 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T11:42:35.959 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:42:36.035 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T11:42:36.036 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T11:42:36.036 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:42:36.135 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T11:42:36.135 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-13T11:42:36.136 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:42:36.221 INFO:tasks.ceph.osd.5:Stopped 2024-09-13T11:42:36.221 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T11:42:36.221 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T11:42:36.221 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:42:36.251 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T11:42:36.251 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T11:42:36.252 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T11:42:36.252 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:42:36.275 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T11:42:36.275 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T11:42:36.275 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:42:36.334 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T11:42:36.335 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T11:42:36.335 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:42:36.356 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T11:42:36.356 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T11:42:36.357 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | head -n 1 2024-09-13T11:42:36.392 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T11:42:36.392 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T11:42:36.625 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T11:42:36.626 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T11:42:36.836 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T11:42:36.836 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T11:42:37.021 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T11:42:37.021 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-13T11:42:37.204 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T11:42:37.205 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-13T11:42:37.421 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T11:42:37.421 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-13T11:42:37.589 INFO:tasks.ceph:Archiving mon data... 2024-09-13T11:42:37.590 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428/data/mon.a.tgz 2024-09-13T11:42:37.590 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-09-13T11:42:37.596 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.WR2qSV0p6m 2024-09-13T11:42:37.597 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.WR2qSV0p6m 2024-09-13T11:42:37.672 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.WR2qSV0p6m 2024-09-13T11:42:37.779 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.WR2qSV0p6m is 97KB 2024-09-13T11:42:37.796 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.WR2qSV0p6m 2024-09-13T11:42:37.802 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428/data/mon.c.tgz 2024-09-13T11:42:37.802 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-09-13T11:42:37.849 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.O9eZpQ7rt5 2024-09-13T11:42:37.850 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.O9eZpQ7rt5 2024-09-13T11:42:37.925 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.O9eZpQ7rt5 2024-09-13T11:42:38.034 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.O9eZpQ7rt5 is 97KB 2024-09-13T11:42:38.049 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.O9eZpQ7rt5 2024-09-13T11:42:38.054 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428/data/mon.b.tgz 2024-09-13T11:42:38.055 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-13T11:42:38.060 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.kYmBb1dVU2 2024-09-13T11:42:38.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.kYmBb1dVU2 2024-09-13T11:42:38.140 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.kYmBb1dVU2 2024-09-13T11:42:38.247 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.kYmBb1dVU2 is 96KB 2024-09-13T11:42:38.262 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.kYmBb1dVU2 2024-09-13T11:42:38.267 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T11:42:38.268 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T11:42:38.270 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T11:42:38.329 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T11:42:38.371 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T11:42:38.371 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T11:42:38.373 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428/remote/smithi005/crash 2024-09-13T11:42:38.374 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T11:42:38.387 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428/remote/smithi039/crash 2024-09-13T11:42:38.387 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T11:42:38.426 INFO:tasks.ceph:Compressing logs... 2024-09-13T11:42:38.427 DEBUG:teuthology.orchestra.run.smithi005:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T11:42:38.434 DEBUG:teuthology.orchestra.run.smithi039:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T11:42:38.460 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28834.log 2024-09-13T11:42:38.460 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29304.log 2024-09-13T11:42:38.460 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29330.log 2024-09-13T11:42:38.460 INFO:teuthology.orchestra.run.smithi005.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.28834.log.gz 2024-09-13T11:42:38.461 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28992.log 2024-09-13T11:42:38.461 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28914.log 2024-09-13T11:42:38.461 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29304.log.gz 2024-09-13T11:42:38.461 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29330.log: /var/log/ceph/ceph-client.admin.28992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27818.log 2024-09-13T11:42:38.462 INFO:teuthology.orchestra.run.smithi005.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29330.log.gz 2024-09-13T11:42:38.462 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-13T11:42:38.462 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2%/var/log/ceph/ceph-client.admin.28914.log: -- replaced with /var/log/ceph/ceph-client.admin.28992.log.gz 2024-09-13T11:42:38.463 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28626.log 2024-09-13T11:42:38.464 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.0.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.27818.log.gz 2024-09-13T11:42:38.464 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29200.log 2024-09-13T11:42:38.464 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28914.log.gz 2024-09-13T11:42:38.464 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29408.log 2024-09-13T11:42:38.464 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29096.log 2024-09-13T11:42:38.464 INFO:teuthology.orchestra.run.smithi005.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.28626.log.gz 2024-09-13T11:42:38.464 INFO:teuthology.orchestra.run.smithi005.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29200.log.gz 2024-09-13T11:42:38.464 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28652.log 2024-09-13T11:42:38.466 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-13T11:42:38.466 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29096.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29408.log.gz 2024-09-13T11:42:38.466 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28261.log 2024-09-13T11:42:38.466 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28095.log 2024-09-13T11:42:38.466 INFO:teuthology.orchestra.run.smithi005.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28652.log.gz 2024-09-13T11:42:38.466 INFO:teuthology.orchestra.run.smithi005.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29096.log.gz 2024-09-13T11:42:38.466 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27942.log 2024-09-13T11:42:38.467 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29278.log 2024-09-13T11:42:38.467 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28095.log.gz 2024-09-13T11:42:38.468 INFO:teuthology.orchestra.run.smithi005.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28261.log.gz 2024-09-13T11:42:38.468 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28888.log 2024-09-13T11:42:38.468 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27942.log: /var/log/ceph/ceph-client.admin.29278.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-13T11:42:38.468 INFO:teuthology.orchestra.run.smithi005.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29278.log.gz 2024-09-13T11:42:38.468 INFO:teuthology.orchestra.run.smithi005.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27942.log.gz 2024-09-13T11:42:38.468 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29356.log 2024-09-13T11:42:38.469 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28520.log 2024-09-13T11:42:38.469 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.29356.log: 89.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.28888.log.gz 2024-09-13T11:42:38.469 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.28274.log 2024-09-13T11:42:38.469 INFO:teuthology.orchestra.run.smithi005.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29356.log.gz 2024-09-13T11:42:38.469 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28390.log 2024-09-13T11:42:38.469 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27941.log 2024-09-13T11:42:38.470 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28274.log: /var/log/ceph/ceph-client.admin.28390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28494.log 2024-09-13T11:42:38.470 INFO:teuthology.orchestra.run.smithi005.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28520.log.gz 2024-09-13T11:42:38.470 INFO:teuthology.orchestra.run.smithi005.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28274.log.gz 2024-09-13T11:42:38.471 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27941.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28390.log.gz 2024-09-13T11:42:38.471 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29018.log 2024-09-13T11:42:38.471 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28494.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27941.log.gz 2024-09-13T11:42:38.472 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28098.log 2024-09-13T11:42:38.472 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29070.log 2024-09-13T11:42:38.472 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29018.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28494.log.gz 2024-09-13T11:42:38.473 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28273.log 2024-09-13T11:42:38.473 INFO:teuthology.orchestra.run.smithi005.stderr:gzip/var/log/ceph/ceph-client.admin.29070.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27692.log 2024-09-13T11:42:38.473 INFO:teuthology.orchestra.run.smithi005.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29018.log.gz 2024-09-13T11:42:38.473 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28098.log.gz 2024-09-13T11:42:38.473 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28273.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29070.log.gz 2024-09-13T11:42:38.473 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28086.log 2024-09-13T11:42:38.476 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27938.log 2024-09-13T11:42:38.476 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27692.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28273.log.gz 2024-09-13T11:42:38.476 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28756.log 2024-09-13T11:42:38.476 INFO:teuthology.orchestra.run.smithi005.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.27692.log.gz 2024-09-13T11:42:38.476 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-13T11:42:38.476 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27938.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28086.log.gz 2024-09-13T11:42:38.477 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28416.log 2024-09-13T11:42:38.479 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28756.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27938.log.gz 2024-09-13T11:42:38.479 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28097.log 2024-09-13T11:42:38.479 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28416.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28756.log.gz 2024-09-13T11:42:38.479 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25024.log 2024-09-13T11:42:38.479 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28546.log 2024-09-13T11:42:38.479 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28416.log.gz 2024-09-13T11:42:38.479 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28097.log.gz 2024-09-13T11:42:38.479 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.25024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27927.log 2024-09-13T11:42:38.479 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29434.log 2024-09-13T11:42:38.479 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28546.log: /var/log/ceph/ceph-client.admin.27927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29174.log 2024-09-13T11:42:38.480 INFO:teuthology.orchestra.run.smithi005.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28546.log.gz 2024-09-13T11:42:38.480 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29434.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27927.log.gz 2024-09-13T11:42:38.480 INFO:teuthology.orchestra.run.smithi005.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.25024.log.gz 2024-09-13T11:42:38.480 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29434.log.gz 2024-09-13T11:42:38.480 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29148.log 2024-09-13T11:42:38.480 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28085.log 2024-09-13T11:42:38.482 INFO:teuthology.orchestra.run.smithi005.stderr:gzip/var/log/ceph/ceph-client.admin.29148.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28093.log 2024-09-13T11:42:38.482 INFO:teuthology.orchestra.run.smithi005.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29174.log.gz 2024-09-13T11:42:38.482 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29226.log 2024-09-13T11:42:38.482 INFO:teuthology.orchestra.run.smithi005.stderr: 89.1% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28085.log.gz 2024-09-13T11:42:38.482 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29148.log.gz/var/log/ceph/ceph-client.admin.28093.log: 2024-09-13T11:42:38.482 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28234.log 2024-09-13T11:42:38.482 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28238.log 2024-09-13T11:42:38.482 INFO:teuthology.orchestra.run.smithi005.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28093.log.gz/var/log/ceph/ceph-client.admin.28234.log: 2024-09-13T11:42:38.482 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T11:42:38.482 INFO:teuthology.orchestra.run.smithi005.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29226.log.gz 2024-09-13T11:42:38.482 INFO:teuthology.orchestra.run.smithi005.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28234.log.gz 2024-09-13T11:42:38.482 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29382.log 2024-09-13T11:42:38.483 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28862.log 2024-09-13T11:42:38.483 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.29382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29044.log 2024-09-13T11:42:38.483 INFO:teuthology.orchestra.run.smithi005.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28238.log.gz 2024-09-13T11:42:38.483 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28862.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29382.log.gz 2024-09-13T11:42:38.484 INFO:teuthology.orchestra.run.smithi005.stderr: 91.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T11:42:38.484 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28862.log.gz 2024-09-13T11:42:38.484 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27773.log 2024-09-13T11:42:38.484 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28940.log 2024-09-13T11:42:38.484 INFO:teuthology.orchestra.run.smithi005.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29044.log.gz 2024-09-13T11:42:38.485 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28730.log 2024-09-13T11:42:38.485 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25142.log 2024-09-13T11:42:38.486 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28940.log: 88.4%/var/log/ceph/ceph-client.admin.28730.log: -- replaced with /var/log/ceph/ceph-client.admin.27773.log.gz 2024-09-13T11:42:38.486 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27934.log 2024-09-13T11:42:38.486 INFO:teuthology.orchestra.run.smithi005.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28940.log.gz 2024-09-13T11:42:38.486 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.25142.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28730.log.gz 2024-09-13T11:42:38.487 INFO:teuthology.orchestra.run.smithi005.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.25142.log.gz 2024-09-13T11:42:38.487 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28442.log 2024-09-13T11:42:38.487 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27934.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-13T11:42:38.487 INFO:teuthology.orchestra.run.smithi005.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27934.log.gz 2024-09-13T11:42:38.487 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29122.log 2024-09-13T11:42:38.488 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25217.log 2024-09-13T11:42:38.488 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.29122.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28442.log.gz 2024-09-13T11:42:38.488 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T11:42:38.489 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29460.log 2024-09-13T11:42:38.489 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29122.log.gz 2024-09-13T11:42:38.489 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.25217.log: /var/log/ceph/ceph.audit.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25217.log.gz 2024-09-13T11:42:38.490 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28280.log 2024-09-13T11:42:38.490 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29460.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21425.log 2024-09-13T11:42:38.491 INFO:teuthology.orchestra.run.smithi005.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T11:42:38.492 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28280.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29460.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28572.log 2024-09-13T11:42:38.492 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:38.492 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph.tmp-client.admin.21425.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28280.log.gz 2024-09-13T11:42:38.492 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21425.log.gz 2024-09-13T11:42:38.492 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28468.log 2024-09-13T11:42:38.493 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27896.log 2024-09-13T11:42:38.493 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28468.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28572.log.gz 2024-09-13T11:42:38.493 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28782.log 2024-09-13T11:42:38.493 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28704.log 2024-09-13T11:42:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25031.log 2024-09-13T11:42:38.495 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27896.log: /var/log/ceph/ceph-client.admin.28782.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28468.log.gz 2024-09-13T11:42:38.495 INFO:teuthology.orchestra.run.smithi005.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.27896.log.gz 2024-09-13T11:42:38.495 INFO:teuthology.orchestra.run.smithi005.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28782.log.gz 2024-09-13T11:42:38.495 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27870.log 2024-09-13T11:42:38.495 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27937.log 2024-09-13T11:42:38.498 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27748.log 2024-09-13T11:42:38.499 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28704.log.gz 2024-09-13T11:42:38.499 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27937.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27870.log.gz 2024-09-13T11:42:38.499 INFO:teuthology.orchestra.run.smithi005.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27937.log.gz 2024-09-13T11:42:38.499 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29252.log 2024-09-13T11:42:38.499 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27798.log 2024-09-13T11:42:38.499 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28808.log 2024-09-13T11:42:38.499 INFO:teuthology.orchestra.run.smithi005.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27748.log.gz 2024-09-13T11:42:38.499 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27798.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29252.log.gz 2024-09-13T11:42:38.499 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27798.log.gz 2024-09-13T11:42:38.499 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28966.log 2024-09-13T11:42:38.504 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29484.log 2024-09-13T11:42:38.504 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28598.log 2024-09-13T11:42:38.504 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28808.log.gz 2024-09-13T11:42:38.504 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.29484.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28966.log.gz 2024-09-13T11:42:38.505 INFO:teuthology.orchestra.run.smithi005.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29484.log.gz 2024-09-13T11:42:38.505 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27844.log 2024-09-13T11:42:38.505 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28678.log 2024-09-13T11:42:38.505 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.27844.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28598.log.gz 2024-09-13T11:42:38.505 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.28678.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27844.log.gz 2024-09-13T11:42:38.505 INFO:teuthology.orchestra.run.smithi005.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28678.log.gz 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25103.log 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.25031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25127.log 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr: 87.8%/var/log/ceph/ceph-client.admin.25103.log: -- replaced with /var/log/ceph/ceph-client.admin.25031.log.gz 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.25127.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.25103.log.gz 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25127.log.gz 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27760.log 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.27760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25055.log 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr:gzip/var/log/ceph/ceph.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27795.log 2024-09-13T11:42:38.506 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.25055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27794.log 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.27795.log: 84.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25055.log.gz 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25079.log 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr: 88.2%/var/log/ceph/ceph-client.admin.27794.log: -- replaced with /var/log/ceph/ceph-client.admin.27795.log.gz 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.25079.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27796.log 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.25079.log.gz 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-mon.b.log: 88.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.27760.log.gz 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.27796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25151.log 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27761.log 2024-09-13T11:42:38.507 INFO:teuthology.orchestra.run.smithi039.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.27794.log.gz 2024-09-13T11:42:38.508 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.25151.log: /var/log/ceph/ceph-client.admin.27761.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25151.log.gz 2024-09-13T11:42:38.508 INFO:teuthology.orchestra.run.smithi039.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27761.log.gz 2024-09-13T11:42:38.508 INFO:teuthology.orchestra.run.smithi039.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.27796.log.gz 2024-09-13T11:42:38.538 INFO:teuthology.orchestra.run.smithi039.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-13T11:42:38.553 INFO:teuthology.orchestra.run.smithi039.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-13T11:42:38.570 INFO:teuthology.orchestra.run.smithi005.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-13T11:42:38.597 INFO:teuthology.orchestra.run.smithi005.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-13T11:42:38.616 INFO:teuthology.orchestra.run.smithi039.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-13T11:42:38.618 INFO:teuthology.orchestra.run.smithi039.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-13T11:42:38.628 INFO:teuthology.orchestra.run.smithi005.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-13T11:42:38.631 INFO:teuthology.orchestra.run.smithi005.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-13T11:42:38.648 INFO:teuthology.orchestra.run.smithi039.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-13T11:42:38.649 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T11:42:38.649 INFO:teuthology.orchestra.run.smithi039.stderr:real 0m0.174s 2024-09-13T11:42:38.649 INFO:teuthology.orchestra.run.smithi039.stderr:user 0m0.497s 2024-09-13T11:42:38.649 INFO:teuthology.orchestra.run.smithi039.stderr:sys 0m0.041s 2024-09-13T11:42:39.076 INFO:teuthology.orchestra.run.smithi005.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-13T11:42:39.077 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T11:42:39.077 INFO:teuthology.orchestra.run.smithi005.stderr:real 0m0.640s 2024-09-13T11:42:39.077 INFO:teuthology.orchestra.run.smithi005.stderr:user 0m1.184s 2024-09-13T11:42:39.077 INFO:teuthology.orchestra.run.smithi005.stderr:sys 0m0.104s 2024-09-13T11:42:39.078 INFO:tasks.ceph:Archiving logs... 2024-09-13T11:42:39.078 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428/remote/smithi005/log 2024-09-13T11:42:39.079 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T11:42:39.319 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428/remote/smithi039/log 2024-09-13T11:42:39.320 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T11:42:39.407 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T11:42:39.418 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T11:42:39.419 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T11:42:39.421 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T11:42:39.511 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T11:42:39.519 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T11:42:39.520 DEBUG:teuthology.orchestra.run.smithi005:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:42:39.522 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:42:39.534 INFO:teuthology.orchestra.run.smithi005.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T11:42:39.534 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================== 2024-09-13T11:42:39.534 INFO:teuthology.orchestra.run.smithi005.stdout:+hv01.front.sepi 67.205.162.81 3 u 46 64 37 0.095 -1.083 1.074 2024-09-13T11:42:39.534 INFO:teuthology.orchestra.run.smithi005.stdout:*hv02.front.sepi 63.231.80.2 3 u 48 64 37 0.107 -1.809 0.993 2024-09-13T11:42:39.534 INFO:teuthology.orchestra.run.smithi005.stdout:+hv03.front.sepi 74.6.168.72 3 u 45 64 37 0.091 -0.488 1.096 2024-09-13T11:42:39.534 INFO:teuthology.orchestra.run.smithi005.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:42:39.535 INFO:teuthology.orchestra.run.smithi039.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T11:42:39.536 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================== 2024-09-13T11:42:39.536 INFO:teuthology.orchestra.run.smithi039.stdout:*hv01.front.sepi 67.205.162.81 3 u 48 64 37 0.076 -1.032 1.184 2024-09-13T11:42:39.536 INFO:teuthology.orchestra.run.smithi039.stdout:+hv02.front.sepi 63.231.80.2 3 u 42 64 37 0.120 -1.471 1.005 2024-09-13T11:42:39.536 INFO:teuthology.orchestra.run.smithi039.stdout:+hv03.front.sepi 74.6.168.72 3 u 50 64 37 0.142 -0.517 1.190 2024-09-13T11:42:39.536 INFO:teuthology.orchestra.run.smithi039.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T11:42:39.536 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T11:42:39.563 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T11:42:39.564 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T11:42:39.613 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T11:42:39.664 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T11:42:39.712 INFO:teuthology.task.internal:Duration was 779.563589 seconds 2024-09-13T11:42:39.712 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T11:42:39.763 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T11:42:39.764 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T11:42:39.766 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T11:42:39.806 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T11:42:39.806 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T11:42:39.807 DEBUG:teuthology.orchestra.run.smithi005:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T11:42:39.854 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T11:42:39.854 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T11:42:39.863 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T11:42:39.864 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T11:42:39.903 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T11:42:39.935 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T11:42:39.935 DEBUG:teuthology.orchestra.run.smithi005:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T11:42:39.974 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T11:42:40.037 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T11:42:40.048 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T11:42:40.048 DEBUG:teuthology.orchestra.run.smithi005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T11:42:40.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T11:42:40.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T11:42:40.097 DEBUG:teuthology.orchestra.run.smithi005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:42:40.112 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:42:40.118 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = core 2024-09-13T11:42:40.129 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2024-09-13T11:42:40.164 DEBUG:teuthology.orchestra.run.smithi005:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:42:40.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:42:40.189 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:42:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:42:40.210 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T11:42:40.220 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T11:42:40.221 DEBUG:teuthology.misc:Transferring archived files from smithi005:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428/remote/smithi005 2024-09-13T11:42:40.221 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T11:42:40.257 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903428/remote/smithi039 2024-09-13T11:42:40.257 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T11:42:40.278 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T11:42:40.278 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T11:42:40.298 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T11:42:40.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T11:42:40.333 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T11:42:40.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T11:42:40.363 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T11:42:40.363 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T11:42:40.366 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T11:42:40.371 INFO:teuthology.orchestra.run.smithi005.stdout: 658463 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 11:42 /home/ubuntu/cephtest 2024-09-13T11:42:40.371 INFO:teuthology.orchestra.run.smithi039.stdout: 658624 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 11:42 /home/ubuntu/cephtest 2024-09-13T11:42:40.373 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T11:42:40.413 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T11:42:40.517 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_5925} duration: 779.5635888576508 flavor: default owner: scheduled_akupczyk@teuthology success: true 2024-09-13T11:42:40.520 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T11:42:40.618 INFO:teuthology.run:pass